Automatic work-element detection: the missing piece in developing intelligent coaching systems for cut-to-length logging machinery

The productivity of cut-to-length machine operators exhibits a significant disparity, with the most productive individuals demonstrating twice the efficiency of their less productive counterparts. This discrepancy is largely attributed to variations in work methods. While supervised training has proven effective in streamlining work methods and enhancing productivity, the availability of forest-machine instructors for supervision is limited. Intelligent coaching systems (ICS) are periodically proposed to address this constraint. ICS are computer-based aids that offer machine operators real-time feedback on their work and guide them on how to rationalize their work. The successful implementation of ICS initially requires the development of systems for automatic work-element detection (AWED). Therefore, this article explores the history, current status


Productivity differences among operators
During field studies, the most productive cut-to-length machine operators have been 40-100% more productive compared to their less productive counterparts (cf.Sirèn 1998;Kärhä et al. 2004;Ovaskainen et al. 2004;Purfürst and Erler 2011;Manner et al. 2016).Moreover, productivity typically improves fastest at the beginning of a career and during younger years.Malinen et al. (2018) predicted harvesting productivity by grouping operators based on their age and experience.According to Malinen et al. (2018), operators achieve their peak productivity in their early 40s, exhibiting approximately 20% higher productivity compared to operators in their 20s.Moreover, operators tend to sustain their peak productivity relatively well over time (Malinen et al. 2018).Purfürst (2010) observed that rookie harvester operators generally double their productivity during the first year but thereafter, the rate of productivity improvement diminishes significantly.Nonetheless, even experienced workers may exhibit a learning curve when faced with changes in their work content.For instance, Björheden (2001) found that highly experienced timber-truck drivers who changed from hauling roundwood to loading-full tree sections showed continuous productivity improvement over a 4-year follow-up period.

Differentiating 'operator effect' from 'working method'
In the forest-work literature, the concept of an operator effect is frequently used.However, there is no consensus within the discipline on the distinction between the operator effect and working method (cf.Björheden 2001;Ovaskainen et al. 2004;Lindroos 2010;Purfürst and Erler 2011;Palander et al. 2012;Häggström and Lindroos 2016).In this paper, psychological, cognitive, and motor abilities are considered as part of the operator effect.These abilities are closely related to an operator's inherent aptitude and are not easily or quickly improved through practice.In contrast, working method encompasses manoeuvres and systematic approaches and strategies to accomplish a given task.Operators' (ineffective) working methods are more changeable than their inherent aptitude.Therefore, these terms are not interchangeable, and it is advisable to differentiate the impact of the working method from the operator effect.
However, it is not entirely clear whether the inter-operator productivity differences in the above cited studies primarily depend on the operator effect, different work methods, or both equally.That said, according to Ovaskainen (2009), a productive working method can increase harvesting productivity by 10-15%.According to the simulation study of Schmiedel et al. (2022) harvesting productivity can differ up to 20% between alternative working methods.Moreover, Ovaskainen and Heikkilä (2007) state that being a highly productive operator does not necessarily require superior abilities but rather motivation and comprehensive work planning.Thus, although the research does not provide a direct answer, we can approximate that 15-20% of productivity differences depend on working methods.

Intelligent coaching system (ICS)
Supervised training rationalizes the working methods of forest machine operators and improves productivity (Lopes and Pagnussat 2017;Pagnussat et al. 2021;Burk et al. 2023).However, the supervision of forest-machine instructors is a limited resource.ICS could potentially address this issue (Palmroth 2011;Dreger et al. 2023).ICS is a computer-based aid that provides machine operators with real-time feedback on their work and guides them on how to rationalize it.While ICS is unlikely to completely replace traditional (human) instructors in the foreseeable future, it can ideally complement them.Traditional instructors can only observe the operator's work for a limited time, and they also pose the risk of the Hawthorne effect or observer bias.Observer bias is when individuals modify their behaviour when being observed (Noland 1959).ICS is chiefly free of these shortcomings, and ideally could also provide instructors with follow-up data (i.e.reference material) on the operator's work.
ICS-alike systems have been studied in the transport sector.These studies indicate that ICS could substantially reduce the risk of operators reverting to old inefficient working methods (af Wåhlberg 2007;Beusen et al. 2009;Huang et al. 2018;Pampel et al. 2018).However, the implementation of ICS requires a more detailed and reliable automatic detection of work elements and methods than what is possible with currently available on-board computing systems in forest machinery, especially for forwarders.Therefore, the initial step in developing ICS for cut-to-length machinery is to create systems for automatic data gathering and work-element detection.

Developing a system for automatic work-element detection (AWED)
The controller area network (CAN), i.a.control signals generated by the operator, provide per se useful information on the forest machine's movements.The time when the engine is running, can be distributed into 'effective working time' and 'other time' based on CAN-bus data (Palmroth 2011;Manner et al. 2016).Today's forest machines save this type of information in standardized format in operational monitoring files (mom-file) which are a part of Standard for Forest machine Data and Communication (StanForD) (Arlinger et al. 2021).That said, AWED requires abundant data processing and additional data sources, especially for forwarders.Tervo et al. (2010) and Palmroth (2011) based the harvester-AWED on the control signals and ordinary measurement data produced during the harvester work.The methodology described in Tervo et al. (2010) and Palmroth (2011) has been a basis when developing the TimberLink performancemonitoring system for John Deere forest machines.However, detailed algorithms of commercial products are often trade secrets and consequently not publicly available.

Standardized AWED is technically realism for harvesters
Nevertheless, commissioning of StanForD during the 2000s enabled standardized automatic data gathering from the harvester work (Arlinger et al. 2021).Strandgard et al. (2013) modelled harvester productivity based on the data extracted from StanForD-files (i.e.stm-files).Strandgard et al. (2013) used time differences between consecutive stm-files to estimate time consumption for each stem (i.e.tree).Because stm-files include log volumes they also enabled determining the productivity.Furthermore, Strandgard et al. (2013) compared automatically gathered dataset against manually gathered control dataset and found no significant differences between the datasets.Although the time labels of stm-files enabled modelling productivity, they did not make AWED possible.
In the current StanForD 2010 stm-files are replaced with a newer file standard named harvested production (hpr).When using hpr-files, harvesters do not only record the point of time but also their own geographical position during each felling cut (Arlinger et al. 2021).Although this is a very useful feature in itself, the same deficiency as with stm-files remains; harvester work cannot be divided into work elements based on a single time label per stem.However, standard hpr-data can be complemented with extension variables.Then, in addition to the felling cut, also events such as crosscut, and tilting-up harvester head can be time-labelled.This feature provides useful data points enabling a basic AWED.Extension variables are tested during field studies (Nordström et al. 2018), but at the time of writing they are not formally part of the StanForD.

Developing forwarder-AWED: a slow process
Although a forwarder's CAN-bus provides data on driving and crane use, it does not enable AWED similarly to a harvester's CAN-bus.The main problem is that standard CAN-bus data per se do not provide any time-labelled datapoints to determine whether the forwarder is loading or unloading, even dividing the forwarding dataset into separate loads is impossible.Nevertheless, operators can use the StanForD-file 'forwarded production' (fpr) to manually report each load's start-and end times, loaded assortments and their (approximated) volumes (Arlinger et al. 2021).But because the operator work is requiring already in itself, an 'operator input' poses a risk for (human) error during data gathering (Manner et al. 2019;Eliasson et al. 2023).Consequently, fpr-files are applicable time-data sources only during short-term field studies.Indeed, ICS should rather decrease the operator's workload than increase it.Therefore, better solutions are needed.
For example, the ability to distinguish between loading and unloading crane cycles enables the identification of other work elements, and also the segmentation of forwarding work (dataset) into separate loads.During loading, an empty grapple is craned out to the pile/log(s) to be grasped, and the grapple now holding log(s) is returned to the load space.Conversely, during unloading, the process is reversed.However, the grapple's status, i.e. whether it holds any log(s), must be detected.Manner (2015) proposed equipping the crane with a scale to detect the grapple's status.This proposal has so far been tested exclusively in a real-time machine simulator (Englund et al. 2020).John Deere's TimberLink system detects/detected the grapple's status based on control signals (Manner et al. 2016).Following a grapple-close command, the grapple is expected to hold logs, while after a grapple-open command, it should be empty.TimberLink also assesses/assessed the grapple's position (inside or outside the load space) using control signals.However, today the grapple's position is more precisely and conveniently accessible from the sensor data.Sensorized forest-machine cranes have been commercially available since the 2010s, and they have been used during field studies to calculate boom-tip paths for both forwarders and harvesters (Bhuiyan et al. 2016;Manner et al. 2017;La Hera and Ortiz Morales 2019).
Geofencing is occasionally proposed as a basis for forwarder-AWED.Forwarder records its geographic position at adjustable intervals, often in every 30 seconds.When speed, distance travelled, and geographical position are recorded at intervals, work elements can theoretically be detected based on geofencing.For example, the AWED system of Strandgard and Mitchell (2015) identified and determined dense GPS-point clouds as polygon-shaped log landing(s).When the speed fell below user defined threshold, either work element 'loading' or 'unloading' was recorded depending on whether the forwarder was inside (→unloading) or outside (→loading) of a log-landing polygon.If the speed exceeded the threshold, the work was classified either as 'driving loaded' or 'driving unloaded' depending on the previous work element (i.e.either loading or unloading).Strandgard and Mitchell (2015) found that forwarder-work elements cannot reliably be detected solely based on geofencing, but additional data source(s) is/are required, e.g.vibration sensor.

Detection of working methods and zones is technically possible
Good working methods are well documented in textbooks and scientific literature (Persson 2013;Ovaskainen et al. 2006;Hartsch et al. 2022;Ovaskainen 2023).Fully sensorized cranes enable recording e.g. the angular displacement of a crane pillar and a crane reach when grasping a tree to be felled or a pile/log to be loaded.Moreover, harvester-head equipped with angle sensor can record stems' felling and feeding directions/angles, and piles' placements and alignments.Thus, in general the detection of working methods and visualizing typical working zones will be readily available once AWED is in place.Strandgard and Mitchell (2015) and John Deere's TimberLink were one of the pioneering works applying AI to AWED.Strandgard and Mitchell (2015) based their system on fully automatic geofencing.Whereas TimberLink uses/used Viterbi algorithm to decode the most likely sequence of hidden states, which then enables recognition of a complete work cycle (Tervo et al. 2010;Palmroth 2011;Manner et al. 2016).

Artificial intelligence (AI) and machine vision open new possibilities
AI, when combined with fully sensorized cranes, has unexplored capabilities in detecting work elements, especially those of the forwarder.During loading, logs are picked-up at approximately constant high (i.e.ground) either side of the forwarder, and loading includes abundant driving during/between crane cycles.During unloading these features are typically vice versa.Thus, AI or more specifically machine learning could possibly use those systematic differences to first identify loading and unloading, and then the whole work cycle (Section 3.2).For instance, Ponsse's Ecodrive detects/detected forwarder-work elements based on systematically recurring control-signal sequences (Manner and Björheden 2017).
Information regarding the loaded assortment(s) per crane cycle, loading stop, or at least per load would be valuable when analyzing forwarder work.However, the current technology employed in forest operations does not offer such real-time information.Nevertheless, the integration of machine vision with machine learning holds the potential to detect handled assortments during loading and/or unloading, thereby addressing this issue and enabling the enhancement of automatic forwarding data gathering with assortment specificity (Jiang et al. 2022;Holmström et al. 2023).For harvesters, assortment information is readily available on a log-wise basis through the bucking system (Sections 3.1 and 5).
AI could also gradually replace number of sensors and solve several challenges and tasks collectively.For instance, machine vision could solve in Section 3.2 described task of detecting whether the grapple is holding any log (Li and Lideskog 2023).Moreover, AI, together with a distance-measuring sensor such as light detection and ranging (lidar), could measure crane reach when grasping a tree to be felled or a pile/log to be loaded, thereby identifying working zones (Section 3.3).Mobile applications based on this type of technology are already used for conducting forest inventory (Willén and Söderberg 2017), and Ponsse ( 2023) is working on a solution for harvesters named "Thinning Density Assistant" using lidar technology to locate trees around the machine.
While AI possesses immense potential, achieving unequivocal identification of all work (time) is unlikely.However, such a comprehensive identification is not even imperative because automated data gathering generates substantial volumes of data, enabling selectivity.This involves excluding intervals where, e.g.work elements or assortments may not have been reliably detected.It is crucial to recognize that surpassing a certain saturation point in data quantity does not yield additional benefits (Manner 2015).Therefore, to improve data quality and reliability, AI should contemplate the removal of indistinct time intervals, even if it results in a smaller dataset.

Providing feedback correctly
Considerations for working conditions are crucial when providing feedback on completed work.Firstly, these conditions significantly impact productivity, and secondly, the adoption of correct working methods and strategies is contingent upon them.Despite the harvester's stem-bucking system providing abundant variables related to working conditions, this crucial aspect is currently overlooked in existing machine monitoring and performance systems.For instance, simply considering the impact of stem volume would notably enhance the utility of feedback (Nuutinen et al. 2010;Manner et al. 2023;Pohjala et al. 2024).Moreover, details such as the volume and assortment of each bucked log and the number of logs per stem are readily available through hpr-files to provide further enhancements (Arlinger et al. 2021).
Again, forwarder is more challenging.While the fundamental impact of spatial assortment (volume) distribution on forwarding productivity is widely acknowledged (Manner et al. 2013;Hildt et al. 2020;Berg 2023), this effect has not been adequately modelled yet.Necessary data (assortments' spatial distribution) would be available post harvest e.g. per stand through hpr-files.Extraction distance is another often proposed factor.Although the definition of extraction distance varies between sources, its effect on forwarding productivity is straightforwardly modellable (c.f.Tiernan et al. 2004;Hildt et al. 2020;Berg 2023).Thus, strategic stand-level knowledge about forwarding productivity is currently limited.However, because forwarding work instructions often offer general guidance (e.g.regarding efficient boom-tip paths, working zones etc.), coaching remains pertinent despite the limited understanding of the underlying factors affecting productivity.
Finally, uniform coaching is unlikely to cater to every operator due to variations in individual aptitude (Section 1.1).For instance, Hartsch et al. (2022) suggested adjusting crane-speed settings based on the operator's skill level.It remains to be seen whether future ICS, with the assistance of AI, can adapt and recognize distinct operator types, discern their strengths and weaknesses, and offer operator-tailored guidance.

Reevaluating the landscape: the role of AWED and ICS in cut-to-length logging machinery in the future
Harvester-AWED has been firmly entrenched for decades, and there is technical readiness for further enhancements.The development of a harvester-AWED does not necessitate substantial investments in sensor technology.Instead, leveraging ordinary data flows, including StanForD extension variables, lays a robust foundation for AWED and, consequently, also for ICS.The absence of a standardized harvester-AWED can be attributed, in part, to labour issues and property rights regarding automatically gathered time data (Metsäteho 2017;Skogen 2017Skogen , 2018a,b;,b;Jungergård and Hultin 2021).That said, if an operator's time data is considered sensitive, only that specific operator may have access to the related feedback (Picco et al. 2023).
In stark contrast, the development of forwarder-AWED is in its infancy.The lack of any generally acknowledged solution prompts us to question whether it is time to abandon traditional approaches and consider novel methods for automating data gathering in forwarding.For instance, AI and machine vision, explored in Section 4, emerge as potential gamechangers in automatic data gathering -not only for forwarders but also for harvesters.
An additional dimension to this discussion arises from the shift in resources towards the development of the next generation of logging systems, including remote-controlled and autonomous forest machines (Lundbäck et al. 2022(Lundbäck et al. , 2023;;La Hera et al. 2024).If these new systems gradually replace the existing machinery, the lifespan of ICS comes into question.The lifespan is not necessarily an issue for harvesters because harvester-ICS would technically be available relatively fast.But the uncertain trajectory of forwarder-AWED raises concerns about forwarder-ICS being short-lived when it eventually emerges.
Nevertheless, it is crucial to distinguish between ICS and AWED.ICS may become obsolete if the logging system undergoes significant changes in the coming decades.However, the technology developed for AWED can maintain compatibility with next-generation machinery.The ability to detect work elements in real-time remains a valuable asset, irrespective of the future appearance of logging machinery, e.g.regarding automation grade etc.Thus, a broader interdisciplinary discussion is imperative, encompassing work science and engineering, to strategically allocate limited research resources.What constitutes a realistic and resource-effective ambition level for ICS, and how does it differ between harvesters and forwarders?These same questions should be posed regarding AWED, prompting us to deliberate on alternative paths for the future of automatic data gathering and work guidance.