Evaluation

We demonstrate that with our embedding-based routing framework, we can generate pedestrian paths that can not only align much closer to elaborate preferences, but also make inputting these preferences much simpler, particularly in contrast to popular path-finding frameworks. To fully leverage our aggregated knowledge base, our experiments are conducted within the boundaries of the city of Cambridge, UK.

First, through the lens of walkability, we illustrate shortcomings of the path-finding frameworks (OSRM, Valhalla, and GraphHopper) discussed in the earlier posts on a number of representative start-target routing scenarios. In particular, we identify routing opportunities ignored by these baselines that offered more pedestrian-friendly paths.

Then, we employ the anchor-based scoring pipeline to generate “general” walkability maps, and analyze their scores to contrast the outputs from different encoder model variants. Furthermore, we conceive four realistic preference objectives that address some of the specific demonstrated shortcomings of the pedestrian profiles used by the open-source frameworks. Separately, these aim to maximize the presence of 1) nature and green spaces, 2) shopping-related areas, 3) historically relevant places, and 4) public safety-inducing factors. We analyze the outputs based on these objectives, both individually and relative to outputs generated based on the general walkability criterion.

Finally, we use both the general walkability and preference-based scores to generate pedestrian paths for the same set of start-target destination pairs. We use these examples to highlight our framework’s ability to generate both walkability-focused and custom preference-based paths, and to do so with significantly simplified preference definitions than the baseline alternatives.

Pedestrian Routing Scenarios

We define a corpus of five realistic start-target routing destination pairs in an urban environment, selected to represent a diverse set of archetypal walking scenarios. Together with these pairs, we present routing solutions generated by the three open-source path-finding frameworks (Valhalla, OSRM, and GraphHopper). We discuss these solutions individually, highlighting their weaknesses, and use them to answer our first research question - what are the shortcomings of existing path-finding frameworks.

Pedestrian Routing Profiles

To generate the paths with the OSRM, we relied on a generic pedestrian profile developed by FOSSGIS (Free and Geographical Information Systems 2025), which is widely used in OSRM implementations. In the case of Valhalla and GraphHopper, the path generation was based on pedestrian routing profiles published directly by the frameworks’ respective developers (Valhalla 2025; GraphHopper 2025). We show that these profiles are: 1) highly complicated to configure, and 2) consider only a narrow selection of strictly segment-related elements, ultimately making them hardly employable by individual users for their specific preferences.

This OSRM routing profile is configured in a several-hundred-line-long Lua file (Free and Geographical Information Systems 2025). It prioritizes pedestrian-designated segments and penalizes U-turns and traffic lights. Furthermore, it determines which nodes directly associated with the segments (such as bollards or gates) are permissible and which are not, and assigns exact speed changes to specific surfaces (such as mud or gravel).

The profile used to generate paths with GraphHopper is defined with multiple YAML and JSON files, as well as GraphHopper’s various internal constants (GraphHopper 2025). Similarly to OSRM, it encodes elementary information about segments (such as surfaces or pedestrian-designated profiles), which are then used to calculate the estimated travel times per segment. These times are then used to calculate the fastest option.

Finally, Valhalla uses a large and complex C++ file to define its pedestrian routing logic (Valhalla 2025). It considers a variety of constraints, such as maximum distances, preferred surfaces, or segment types, and penalizes elements such as steps, crossings, or elevation. It also favors pedestrian paths and sidewalks while discouraging alleys or driveways.

Example 1: Supermarket-adjacent Walk

The first exemplary problem we employ is defined by a start destination close to Cambridge’s city center, and a target in a slightly more remote supermarket-adjacent area. Both points are situated in residential areas, and are specific by a proximity to highly walkable spaces (particularly parks).

alt text

Nevertheless, none of the generated solutions takes advantage of the mentioned green spaces. OSRM and GraphHopper both follow a similar route following a pedestrian footpath leading through a nearby shopping mall, and then an adjacent residential area.

Valhalla, in contrast, sticks to sidewalks attached to busy major roads, resulting in a more straightforward solution. Although simple to follow, it does not seek to maximize the pedestrian experience. We hypothesize this is driven mainly by Valhalla’s minimization of infrastructure features (such as junctions and turns) with its penalty costing function.

Example 2: Long City-spanning Walk

The second routing example defines a problem of substantially broader spatial proportions, stretched between two remote destinations at opposite ends of Cambridge. Here too is the problem’s space is largely defined by residential areas, but presents an even greater opportunity to leverage greenery towards maximizing walkability.

alt text

However, the solutions generated by the open-source frameworks carry much resemblance to those in the previous problem. Again, OSRM and GraphHopper utilize mostly paths and sidewalks in residential areas, and do not pursue routing opportunities presented by the parks. Nonetheless, in this regard, Valhalla seems to perform slightly better, incorporating significantly more greenery into its output. However, considering its costing mechanisms, we attribute this (again) to Valhalla’s preference towards avoiding complicated infrastructural features, such as junctions or crossings (as is also apparent from the distinctly simple shape of its route).

Example 3: Greenbelt Walk

In our third example, we bring the routing scenario closer to dense urban environments - between a historical university college in the city center and another, slightly more remote, college. Here, the routing algorithms are provided with the option to either align their solutions with the walkable but remote playing fields and parks, or leverage the historical and pedestrian-friendly city center.

alt text

The considered open-source frameworks mostly leverage the first option. Even though the OSRM-generated route delves slightly further into the historical center, it later joins the other two outputs in exploiting the direct but remote footpaths and sidewalks outside of the urban center. This is because none of these frameworks implements a mechanism with the ability to recognize or reward subjectively enticing environments, such as Cambridge’s historical center.

Example 4: City Center Walk

The fourth example in our set of routing scenarios presents a dense urban environment, with many opportunities for specific user preferences, invocable by, for example, various shopping areas, restaurants, or museums.

alt text

The open-source frameworks, however, allow for no option to input these preferences into their path generation. As illustrated by this routing scenario, the paths subsequently result in plain solutions, maximizing no specific objective that could be pursued in an environment of this kind.

Example 5: Suburban Stretch Walk

In the final routing scenario, we utilize a start-target destination pair in a deeply residential area on the outskirts of Cambridge. We include this example to highlight the importance of subtle nuances, particularly in segments’ busyness, that are relatable to important pedestrian factors, such as the feelings of public safety.

alt text

This scenario produces another showing of divergence between the solutions generated by GraphHopper and OSRM, and Valhalla. Once again, we attribute this to Valhalla’s inclination towards spatially straightforward routes, which is reflected in a path that simply follows a singular road with no turns.

References

← Previous: Walkability Chapter 4: Design and Implementation
Next: Walkability Chapter 5: Evaluation (Part 2: Walkability Assessment) →