Spatiotemporal localisation patterns of technological startups: the case for recurrent neural networks in predicting urban startup clusters

Although there have been many attempts in the literature to standardise the definition of “startup” (e.g. Nauman and Edison 2010; Reisdorfer-Leite et al. 2020), there is still no consensus about the description of this term. In their mapping study, Paternoster et al. (2014) identified several themes which dominate the literature on technological startups. According to the papers they screened, startups are usually defined as new companies that lack resources, have a small, inexperienced team of employees, develop mainly one innovative product and evolve rapidly in a very uncertain environment (Paternoster et al. 2014). However, depending on the focus of a particular study, different factors were taken into consideration when deciding whether a company is a startup or not. Thus, this study adopts the following definition of a technological startup—it is a newly founded business entity (operating for up to 5 years) whose main specialisation is in the domain of computer technology and software.

Specifically, the total sample consists of 11′100 business entities established between 2010 and 2018 in capital city of Warsaw, Poland. The period of analysis was chosen to fit between the global financial crisis and the COVID shock, providing a long window of stable spatial pattern development. The companies were selected concerning their specialisation statement. The firms that will be included in the study specialise in manufacturing computers and their peripherals, producing electronic and optical products, publishing of computer games, computer programming, software-related consultancy, or conduct other information technology and computer service activities. Company information was obtained from the ORBIS database and address information was geocoded with the usage of MapQuest API (MapQuest 2018).

The choice of city for this analysis was driven by several factors. Firstly, while there is abundance of research on innovative businesses in developed economies such as the USA, the United Kingdom or Germany (e.g. Banal-Estañol et al. 2019; Florida and Mellander 2017; Geibel and Manickam 2016; Pisoni and Onetti 2018), a very limited amount of papers has been devoted to the emerging markets of Central-East Europe. Secondly, the Warsaw startup community is growing rapidly and attracting larger sources of funding from both domestic and foreign capital. After much anticipation, the first “unicorn”, a startup estimated to be worth more than 1 billion USD, was announced in 2021 (Bełcik 2021). The presence of “unicorns” is a signal to investors that the market is highly attractive and profitable (Suwarni et al. 2020). Thirdly, Warsaw, with a population of 1′794′200 and an area of 517.24 km² (Urząd Statystyczny w Warszawie, 2021), is one of Europe’s metropolises. Being a highly heterogeneous city, with areas of business concentration and less populated peripheries, it creates an interesting case study for different location opportunities and consequences of such choices for startups. Fourthly, Warsaw has its own special circumstances, which makes it a great area for location studies. Unlike most American cities, Warsaw does not have strict plans that dictate where and where not to locate a business. This makes business location truly dictated by the factors unique to startup founders. Additionally, Warsaw’s urban organisation has gone through dramatic changes over the last century. After being almost completely destroyed in 1944 and rebuilt by the communist government, Warsaw’s urban planning has undergone significant transformation over the past decades, which is unusual for other cities in Europe and beyond. Moving forward in time, a new, modern Warsaw with a vibrant business centre has only emerged in the last few years, after the financial crisis and even more so after the Brexit shock (Fig. 1). Inspiring instances of regeneration and rapid construction of new high-rise buildings continue, creating new attractive business areas that attract companies and change the fabric of the city as a whole. These specific conditions of Warsaw make it an even more interesting case for spatial research, showing how forces previously studied for already developed markets operate in a dynamic, almost unlimited environment. As a result, the conclusions drawn for Warsaw can be successfully applied to other cities around the world.

Following the call for analysis of entrepreneurial activity at a “very low level of aggregation” suggested by Guzman and Stern (2016), the localisation choices of technological startups will be here investigated from the micro-geographic perspective. Individual localisation points will be considered, created by geocoding the business address with an accuracy of 0.1 m¹—such precision of geocoding allows for accurate inference based on the methods used later. The highest aggregation level that will be used is a 1 km per 1 km grid, the size of which is dictated by the availability of a population grid derived from the census (Portal Geostatystyczny 2021).

The empirical strategy of this paper is organised in three stages: recognition of startup concentration in urban space, identification of clusters based on the density of startups at the single company level, and prediction of future cluster localisations with 1 km precision. The main objective, which is to verify if localisation patterns consistent with sharply attenuating agglomeration externalities can be observed, will be achieved with a mixture of ML and deep learning methods² (Fig. 2).

Before moving on to the empirical results, it would be useful to provide some basic description of how neural networks work and why they can be a beneficial tool for regionalists. Therefore, this section will outline the most important concepts related to neural networks (NNs).

When describing how a neural network works, it is important to start with the very core of this model, which is the single neuron, developed from the inspiration of the brain's neuronal cells in 1943 (McCulloch and Pitts 1943). A neuron, in mathematical terms, is a function that takes an input, transforms it with an appropriate weight, applies a (activation) function to the resulting output, and then passes this output on. The goal of this transformation is to match the final output of the network to the true labels of the input data (like the y-variable in OLS). A single neuron (a so-called perceptron) forms the most basic structure of a neural network model (Fig. 4). Note that by setting the activation function g as sigmoid (logistic) in the form \(\sigma \left(z\right)= \frac{1}{1+{e}^{-z}}\), the model is equivalent to a logistic regression (Lee and Almond 2003).

The output of a single neuron is represented by the following equation:where g is the activation function, w₀ is a constant, d is the number of input variables, w_i is a weight applied to a given input, and x_i is a value of a given input. The neuron takes d input variables \({x}_{i}\), which are transformed with weights \({w}_{i}\) to match the learned pattern. The weights are calibrated during the model training. A weight \({w}_{0}\), or a constant, is added to account for possible additional influences on the data that are not explained directly by the inputs. The activation function g is applied to the result of input weighting. Various forms of activation function are used in practice, e.g. binary, linear, sigmoid (logistic), tanh (hyperbolic tangent), softmax, ReLU, etc. (Sharma et al. 2017). The form of the activation function for each network layer is set according to previous use cases of similar network structures for similar contexts. The activation functions can also be reparametrised in search of hyperparameters along with the number of neurons in the final finetuning of the model.

Since a single neuron only recognises linear patterns in the data, neurons can be combined to form layers (Fig. 4). A layer in neural network can be understood as running a set of simultaneous regressions, the individual results of which are then combined (flattened) to provide the final output of the network. In more complex settings, where patterns in the data are more nuanced, it is common to stack several layers together, creating a deep neural network (Fig. 4C). Usually the layers are fully connected—meaning that the outputs from each neuron from the previous layer are passed as inputs to each neuron in the next layer. This mechanism allows the model to catch nonlinear complex patterns in the data with maximum efficiency.

A neural network “learns” the patterns in the data by shifting its weights (which is similar to fitting coefficients in OLS). The procedure of weights shifting is an iterative process, where an error measure (loss) is reported at each stage. The error measure is feedback to the model about how distant its predictions are from the true data labels (or from the true values of the dependent variable). Error measures are matched to the characteristics of the data—for predictions of continuous values, the root-mean-squared error (RMSE—commonly used in OLS evaluation) can be used, while for predictions of categorical variables, an entropy-based measure such as binary cross-entropy is usually used.

The pace at which the model reshuffles its weights is dictated by the learning ratio parameter. The higher the learning rate, the greater the changes in weights with each iteration. With each learning iteration (epoch), the learning rate should decrease, making the model more cautious about future changes in the weights. This behaviour is ensured by the specification of learning rate annealing, a parameter responsible for adjusting the learning rate as changes in the loss function become flatter.

Time series data provides its own set of challenges for NN modelling. In the standard setting, the NN model assumes that each input provided is independent of the others. Using this assumption, the model attempts to understand the pattern for each input individually. For time series data, it is important to capture the connections between data points that result from their sequential nature. This challenge is solved by recurrent neurons, which do not simply pass learned information to the next layer (a feedforward mechanism), but train on the output of one time step to find a connection to the next. By separating the information from the sequence and examining the pattern at each time step, the network is able to "retain memory" of previous events while considering subsequent events. Variations of this “memory” mechanism are used in recurrent neural networks (RNN), long short-term memory Loss (LSTM), and many others. The use of recurrent neurons allows the model to learn temporal patterns even on short time series. This ability, combined with the recognition of non-linear data patterns, makes RNNs much more effective in time series forecasting than any “traditional” econometric approaches.

Neural networks, like many other ML techniques, are often described as black box models. This means that it is not possible to easily look at their final structure and understand the (fuzzy) relationships that have been discovered in the data. Compared to models such as OLS—the most well-known white-box—black-box models do not offer the possibility to explain the phenomenon they model. This situation is slowly changing with the emergence of new developments in explainable artificial intelligence (XAI). However, there is still much room for improvement in this area. Thus, ML techniques are not a competition to econometric models, they are simply another statistical approach that focuses more on predictive power than explanatory power. Keeping in mind this divide between predictive vs. interpretability and linear relationships vs. fuzzy patterns, one can successfully draw from both worlds and improve their research.

Accounting for the spatial dimension in ML or NNs is a growing area of research that is still quite underdeveloped. An excellent overview of the current state of the art in spatial machine learning can be found in Kopczewska (2022). Fitting into the same theme, this paper aims to show the usefulness of NNs in regional science and presents a fresh perspective on machine learning methods in spatial analysis.

The first part of the formal analysis will consider the startup concentration in general. It will be investigated whether any significant hotspots have emerged and if such trends are evident in annual samples or only in the whole sample.

Figure 5 shows the result of kernel density estimation for the total sample. It appears that startups, in general, are localising densely on the left side of the Vistula. There is some concentration on the eastern side of the map, but it is much smaller than that identified on the western side. One very visible hotspot can be observed in the city centre, where startup concentration is significantly higher than in other parts of the city. However, three other concentration areas in the southern part of the city can be also observed. These are less concentrated than the one in the centre, but are indeed present. The startup localisation structure in Warsaw is polycentric and not homogeneously distributed in the urban space.

Tracking the yearly subsamples of newly created startups, it can be observed that the concentration pattern changed over time (Fig. 6). The most visible tendency observed in the yearly subsamples is the consolidation of central hotspots. While in 2010, only some general concentration of startups on the left side of the Vistula River could be observed, in the following years, startups more intensively chose central hotspots rather than the western part of the city. Between 2016 and 2018, the importance of the central hotspot grew, until much of the startup concentration was localised in the most central, prestigious city area.

The kernel density estimation results show that technological startups tend to co-locate. The strength of the concentration trend is changing over time, but the conclusion remains constant—the localisation pattern in each year showed significant hotspots of startup concentration. In these hotspots, the companies are located densely, and their concentration is much higher than in the rest of the city. These results suggest that technological startups may follow spatial patterns consistent with highly localised externalities operating in cities. However, to assess the strength of the concentration trend, it is necessary to compare the number of co-located and dispersed companies within the city area. The DBSCAN method will be used for this purpose, and the results will be discussed in the next section.

DBSCAN is a method that can track even small groupings of companies formed within a target distance from each other. Such an analysis can show how many clusters are created, their sizes and how many companies locate outside of the business groupings. It allows tracking the information omitted by the previous method and verifying the relative strength of the concentration and dispersion trends.

First, it will be checked how many clusters can be identified in the entire sample of startups. With the parameters eps = 0.0035⁴ and minPts = 20, the algorithm recognised sixty clusters (Fig. 7). Their sizes varied between 4625 to 9 observations in each. The largest cluster (dark red points located centrally on Fig. 7) was localised in the city centre, on the west Vistula’s bank and coincided with the hotspot identified by the previously discussed method. Despite the identification of that many groupings, there is also a large share of companies that locate outside clusters (Fig. 7). In the total sample, the method has recognised 8288 such firms, which represent 74.7% of the analysed sample.

The number and size of identified clusters differed throughout the years (Fig. 8). In 2010, of 821 startups founded, only 30% were localised in nine clusters (Table 1). The remaining companies were dispersed in the urban space with a slight preference for locations in the western part of the city. In 2011, due to the lower number of founded companies, only two clusters were identified—one of which was located directly in the city centre. In the 2012–2016 samples, a growing number of groupings were recognised, with an increasing size and significance of the central cluster. The share of startups located outside of any grouping was decreasing—from 52.9% in 2012 to 21.2% in 2016 (Table 1). However, in 2017 and 2018, this pattern seemed to change. Fewer clusters were identified, and most of the grouped companies were located in the city centre, concentrating in much smaller areas than before. Simultaneously, the share of dispersing companies increased to the levels identified in 2014 (the share of companies dispersing in the urban area was 35.1% in 2017 and 36.4% in 2018). However, while more companies decided to follow the dispersion trend, the concentration of remaining businesses was stronger than ever. Clustering tendencies were much more localised, following a pattern consistent with agglomeration externalities operating at fine spatial scales.

Some changes have been observed in the spatiotemporal localisation pattern of technological startups. Over the years, the concentration trend has strengthened. Initially, this tendency was evident in the increasing share of clustering startups and then in the increasing density of the newly formed groupings. While the share of clustering startups fluctuates, the concentration trend is relatively stronger than the dispersing tendency. Clusters created by startups are dense and concentrated in similar localisations—following a polycentric pattern with hotspots located on the west side of the city. Following this insight, it can be concluded that most technological startups form a spatiotemporal localisation pattern based on concentration and co-location. These two elements, binding companies together in small distinct city areas are consistent with highly localised agglomeration effects. Thus, it can be suspected that small-scale agglomeration externalities may be important forces influencing the intra-urban location decisions of most technological startups.

Knowledge of trends in startup localisation patterns can be very beneficial. Being able to predict future hotspots can be helpful in accurately planning urban infrastructure, locating startup support facilities or proposing local programmes promoting entrepreneurship in less preferred areas. Although the theoretical roots of intra-urban startup location decisions are not yet fully understood, it is possible to utilise deep learning methods such as NNs to “learn” the nonlinear patterns available in the data.

The neural network model presented in this paper works very well in predicting the localisation of startup clusters in Warsaw, Poland. It can also provide valuable information about general trends in the popularity of specific city areas at the very detailed scale of 1km² grid cell. Powered by the current data, the model can be a very valuable tool for policy-making. However, these results are not limited to this one specific city. Following this successful model structure, it can be easily recalibrated to make predictions for another metropolis. As shown in the article, even short panel data can be used in this approach. In this case, having data on the addresses of newly founded startups across nine years was enough to build a model that predicts innovation clusters with more than 95% accuracy.

Additionally to their methodological value, the results from this section also contribute to the main thesis of the paper. The persistence of the concentration trend of technological startups allows for accurate predictions even with a considerably simple modelling structure. While there have been changes in the size of clusters and their exact location over the years, it turns out that a 3-year sequence of the attractiveness of an area is sufficient to decide whether a new cluster will be formed there in the upcoming year. Moreover, the effect of including the spatial feature is so effective due to the co-localising trend. When startups locate densely in one location (to utilise the small scale agglomeration externalities), the density of startups is likely to be high in the direct neighbouring areas as well (because externalities spread across space). The co-localising tendency discovered in the previous sections improves the stability of the model results, showing that the clustering process is highly dependent on micro-geographical trends appearing in the neighbouring area. Accurately predicting future hotspots of technological startup activity is possible because these companies benefit from the highly localised externalities that require them to locate near areas that have already proven attractive to startups in previous years.