Artificial intelligence-aided railroad trespassing detection and data analytics: Methodology and a case study

https://doi.org/10.1016/j.aap.2022.106594Get rights and content

Highlights

  • Trespassing is a safety concern for highway users and railroads at highway-rail grade crossings and along rights-of-way.

  • A novel vision-based trespassing detection technology was developed to automatically detect trespassing events and collect basic characteristics.

  • This practice-ready tool was validated and applied with over 1,600 h of video data from one grade crossing.

  • Data-driven practical trespassing safety risk mitigation solutions were proposed from engineering, enforcement, and education perspectives.

Abstract

The railroad industry plays a principal role in the transportation infrastructure and economic prosperity of the United States, and safety is of the utmost importance. Trespassing is the leading cause of rail-related fatalities and there has been little progress in reducing the trespassing frequency and deaths for the past ten years in the United States. Although the widespread deployment of surveillance cameras and vast amounts of video data in the railroad industry make witnessing these events achievable, it requires enormous labor-hours to monitor real-time videos or archival video data. To address this challenge and leverage this big data, this study develops a robust Artificial Intelligence (AI)-aided framework for the automatic detection of trespassing events. This deep learning-based tool automatically detects trespassing events, differentiates types of violators, generates video clips, and documents basic information of the trespassing events into one dataset. This study aims to provide the railroad industry with state-of-the-art AI tools to harness the untapped potential of video surveillance infrastructure through the risk analysis of their data feeds in specific locations. In the case study, the AI has analyzed over 1,600 h of archival video footage and detected around 3,000 trespassing events from one grade crossing in New Jersey. The data generated from these big video data will potentially help understand human factors in railroad safety research and contribute to specific trespassing proactive safety risk management initiatives and improve the safety of the train crew, rail passengers, and road users through engineering, education, and enforcement solutions to trespassing.

Introduction

Based on statistics from the Federal Railroad Administration (FRA) of the United States (U.S.) Department of Transportation, the U.S. railroad system is comprised of approximately 830 railroads, 134,000 miles of track, and 210,000 railroad crossings (FRA, 2018a). Trespassing accidents along rights-of-way (ROW) and at highway-rail grade crossings constituted over 90% of rail-related deaths over the past ten years (FRA, 2018a). More specifically, there were 855 trespass-related fatalities in 2017, which demonstrated an increase of 18 percent from 2012 (FRA, 2018b). In addition to fatalities, these incidents resulted in other serious consequences, such as nonfatal injuries, train derailments, hazardous material spillage, train delays, and traffic congestion. From 2012 to 2016, trespassing accidents in the United States cost railroads and society approximately $43 billion (FRA, 2018b), a sum that did not cover indirect costs (e.g., emotional distress or productivity losses). The FRA (2016a) concluded that most trespassing deaths occurring each year are preventable if effective countermeasures were implemented.

Amongst the limited studies of railroad trespassing, most researchers encountered challenges due to limited data resources and uncertain data quality. Most publicly available trespassing data takes the form of casualty information or grade crossing accidents, and does not include near-miss events. However, the FRA (2018b) postulated that the number of trespassing occurrences each year far exceeds the number of fatalities and injuries and more data on trespassing events that do not result in casualties would be valuable to railroad safety researchers. In other words, while the accident reports submitted to the FRA by railroads have proven to be helpful to railroad researchers, most of the valuable data on trespassing is still missing. Trespassing events indicate certain behaviors that may lead to severe consequences if they occur repeatedly. Specifically, the near-miss events of trespassing, involving common causation against trespassing accidents, can contribute to developing the perceptions of trespassing risks with a sufficient number of events. Learning from these trespassing events is critical towards better education for the public on trespassing safety, law enforcement, and engineering solutions to prevent trespassing on railroad tracks. The increasing availability of video data in the rail industry makes the collection of trespassing data more feasible.

Deployment of camera systems continues to increase in the United States following the 2015 Fixing America’s Surface Transportation (FAST) Act, which mandated the installation of cameras throughout passenger rail lines to promote safety objectives (FRA, 2015). In addition, the Transportation Security Authority (TSA) provided funding for surveillance in transit and passenger rail areas (Elias et al., 2016). Cameras can be found throughout rail lines, yards, bridges, grade crossings and stations, which provide numerous video data sources for railroads. However, most camera systems are reviewed manually by railroad crew, train police, or local police which is labor-intensive and expensive. Limited resources and operator fatigue (Dee and Velastin, 2008) can potentially lead to missing trespassing events. Besides, the trespassing incident/accident risks along the right-of-ways and at grade crossings are challenging to monitor and to manage since they involved non-railroad personnel (e.g., pedestrian, vehicle drivers). To address these challenges and leverage the untapped potential of this big video data, this research develops a novel Artificial Intelligence (AI)-aided tool that is capable of localizing and identifying trespassing events in both archival video data and live streams with acceptable processing speed and accuracy. You Only Look Once (YOLO), an emerging object detection algorithm developed by Redmon et al. (2016), Redmon and Farhadi (2018), is utilized in the trespassing detection methodology to achieve high-accuracy trespassing detection with relatively low computation cost. With this practice-ready technology, over two months of video data from one grade crossing are processed and over 3,000 trespassing events are detected and analyzed in this study. These detected events, along with recorded trespassing video clips, can contribute to developing practical trespassing risk mitigation strategies and improving the safety of the train crew, rail passengers, and road users.

Section snippets

Trespassing on railroad property

Railroad trespassing is defined as an event when any unauthorized person or vehicle enters or remains on a railroad right-of-way, grade crossing, equipment, or facility (FRA, 2018b). Railroads own their rights-of-way and have a reasonable expectation of operating on their property without the presence or interference of unauthorized persons. Pedestrians and motorists are only permitted on railroad property where a roadway intersects with the railroad tracks at the same level or grade, provided

Overview of You Only Look Once (YOLO)

YOLO uses features learned by a single deep convolutional neural network to detect objects. As introduced in the previous section, most deep learning-based object detection algorithms, such as the R-CNN family, have a complex detection pipeline, in which bounding box generation, object classification, duplicate detection elimination, and bounding box refining and rescoring are executed sequentially. Instead, YOLO sees the entire image or video frame and implicitly encodes contextual information

Overview of selected grade crossing

To validate the functionality of the proposed AI-based trespassing detection technique, a grade crossing located in New Jersey is selected as a case study, although the developed methodology can also be applied to rights-of-way. The selected crossing experiences about 110 activations per day, with the majority being commuter trains. One train station with three parking lots, two to the west of the train tracks and one to the east, are adjacent to the grade crossing. Several restaurants,

Conclusions

This paper presents a state-of-the-art AI-aided methodology with high-accuracy fast-processing railroad trespassing detection capabilities for both highway-rail grade crossings and rights-of-way. The applications of YOLO and computer vision in trespassing detection have been validated in around 1,632 h of videos with reasonable accuracy. Around 3,000 trespassing violations are detected and recorded during the analyzed period. In the location-specific case study, the collected trespassing

Future work

Firstly, future work would focus on accuracy improvement by mitigating noise from sunlight on the surface of red signal and extreme weather conditions. For example, the automatic identification of grade crossings gate position can be used as a supplemental activation trigger. Moreover, analyses of passive, non-signalized grade crossings can also be explored in the future. Secondly, future work can investigate the possibility of integrating a proposed AI-based trespassing detection tool with

CRediT authorship contribution statement

Zhipeng Zhang: Methodology, Validation, Formal analysis, Investigation, Data curation, Writing – original draft. Asim Zaman: Validation, Investigation, Data curation. Jinxuan Xu: Conceptualization, Methodology, Supervision, Writing – original draft, Project administration. Xiang Liu: Investigation, Methodology, Project administration, Writing – review & editing.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Acknowledgements

The authors thank Mr. Francesco Bedini Jacobini from the FRA, Mr. Marco daSilva from the Volpe National Transportation Systems Center, and Mr. Todd Hirt from the NJDOT for their feedback and insight during this research. The first author was financially supported by the Rutgers, The State University of New Jersey in the preparation of this manuscript. The views and opinions expressed herein at these of the authors.

References (46)

  • J. Zhang et al.

    A real-time Chinese traffic sign detection algorithm based on modified YOLOv2

    Algorithms

    (2017)
  • M. Zhang et al.

    A comparative study of rail-pedestrian trespassing crash injury severity between highway-rail grade crossings and non-crossings

    Accid. Anal. Prev.

    (2018)
  • Z. Zhang et al.

    Automated detection of grade-crossing-trespassing near misses based on computer vision analysis of surveillance video data

    Saf. Sci.

    (2018)
  • S. Arabi et al.

    A deep-learning-based computer vision solution for construction vehicle detection

    Comput.-Aided Civ. Infrastruct. Eng.

    (2020)
  • A. Catalano et al.
  • P. Chakraborty et al.

    Traffic congestion detection from camera images using deep convolution neural networks

    Transp. Res. Rec.

    (2018)
  • S.G. Chase et al.

    Effect of Gate Skirts on Pedestrian Behavior at Highway-Rail Grade Crossings

    (2013)
  • Congressional Research Service (CRS), 2018. Positive train control (PTC): overview and policy issues. R42637,...
  • DaSilva, M.P., Baron, W. and Carroll, A.A., 2012. Highway rail-grade crossing safety research: Railroad infrastructure...
  • H.M. Dee et al.

    How Close Are We to Solving the Problem of Automated Visual Surveillance?

    Mach. Vision Application

    (2008)
  • B. Elias et al.

    Transportation security: issues for the 114th congress

    Library of Congress, Congressional Research Service

    (2016)
  • Federal Railroad Administration, 2008. Compilation of pedestrian safety devices in use at grade crossings. Washington...
  • Federal Railroad Administration, 2015. The Fixing America’s Surface Transportation Act (P.L....
  • Cited by (13)

    • Real-time dense traffic detection using lightweight backbone and improved path aggregation feature pyramid network

      2023, Journal of Industrial Information Integration
      Citation Excerpt :

      For instance, the objects of a person and train are quite difficult to be detected using a traditional detector designed for street vehicles in the same image. Unfortunately, there are more frequent traffic delays and accident reported recently that are related with the railroad grade crossing area where mixed traffic is present [11, 12]. Additionally, it needs to be mentioned that the accurate detection of train and pedestrians are the basis of efficient traffic management and safety assurance when there is railroad involved in the urban areas.

    • Quantitative risk assessment of railway intrusions with text mining and fuzzy Rule-Based Bow-Tie model

      2022, Advanced Engineering Informatics
      Citation Excerpt :

      Specifically, the images of unauthorized objects can be obtained through stationary cameras, train-mounted cameras, or unmanned aerial vehicles [10]. Then, traditional computer vision approaches [11] and deep learning models [12,3] assist the video surveillance. LiDAR data is another useful type of information that provides precise depth information and complements camera data well [13].

    • Design of Indoor Navigation Scheme Based on Bluetooth Low Energy

      2024, Lecture Notes in Electrical Engineering
    View all citing articles on Scopus
    View full text