literature review machine vision

We apologize for the inconvenience...

To ensure we keep this website safe, please can you confirm you are a human by ticking the box below.

If you are unable to complete the above request please contact us using the below link, providing a screenshot of your experience.

https://ioppublishing.org/contacts/

Advertisement

Review of current vision-based robotic machine-tending applications

• Critical Review
• Published: 15 February 2024
• Volume 131 , pages 1039–1057, ( 2024 )

Cite this article

• Feiyu Jia 1 ,
• Yongsheng Ma 2 &
• Rafiq Ahmad   ORCID: orcid.org/0000-0002-2290-5885 1  

527 Accesses

Explore all metrics

The manufacturing sector is a fundamental pillar of worldwide economies, contributing markedly to global economic growth. However, the manufacturing industry is persistently confronted with issues impeding its development and expansion, such as manpower shortages, safety concerns, high initial investment for installation, and long return on investment. Within this context, machine tending has become a crucial component of the manufacturing process and potentially serves as a viable solution to the afore-mentioned predicaments. Over the past 5 years, implementing automated machine-tending systems has widely extended from simulation or laboratory environments to practical applications in manufacturing workshops as robotics and artificial intelligence develop rapidly. To fully benefit from the potential of machine-tending applications, it is necessary to comprehend and tackle its associated challenges. Therefore, this paper aims to contribute to the evolution of machine-tending applications by investigating the impacts of emerging trends of advanced technologies, such as autonomous mobile robots, computer vision, machine learning, and deep learning. This systematic literature review is based on the Protocol of Preferred Reporting Items for Systematic Review and Meta-Analyses to analyze the 50 scientific literature related to machine tending in the last five years.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Subscribe and save.

• Get 10 units per month
• Download Article/Chapter or eBook
• 1 Unit = 1 Article or 1 Chapter
• Cancel anytime

Price excludes VAT (USA) Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Rent this article via DeepDyve

Institutional subscriptions

Similar content being viewed by others

Deep Learning in Robotics for Strengthening Industry 4.0.: Opportunities, Challenges and Future Directions

Digital Transformation in Smart Manufacturing with Industrial Robot Through Predictive Data Analysis

Object Classification in an Intelligent Robotic Cell Using Deep Learning

Explore related subjects.

• Artificial Intelligence

Data availability

No data was used for the research described in the article.

Chains RS, The IN, Era NEW, Manufacturing OF (2022) The 2022 World Manufacturing Report

Helper S, Krueger T, Wial H (2021) Why does manufacturing matter? Which manufacturing matters? A policy framework. SSRN Electron J:1–53. https://doi.org/10.2139/ssrn.3798089

ILO (2022) “ILO Monitor on the world of work . Ninth edition,” pp 1–25

Rooks B (2003) Machine tending in the modern age. Ind Robot 30(4):313–318. https://doi.org/10.1108/01439910310479568

Article   Google Scholar  

Gupta SK (2020) Towards next generation pandemic proof factories. Smart Sustain Manuf Syst 4(3):260–263. https://doi.org/10.1520/SSMS20200058

Article   MathSciNet   Google Scholar  

Lee J, Davari H, Singh J, Pandhare V (2018) Industrial Artificial Intelligence for industry 4.0-based manufacturing systems. Manuf Lett 18:20–23. https://doi.org/10.1016/j.mfglet.2018.09.002

Li D, Huang Y (2008) “A general robot application platform for machine tending,” 2008 IEEE International Conference on Robotics, Automation and Mechatronics, RAM 2008, pp 1–5.  https://doi.org/10.1109/RAMECH.2008.4690867

Iglesias I, Sebastián MA, Ares JE (2015) Overview of the state of robotic machining: current situation and future potential. Procedia Eng 132:911–917. https://doi.org/10.1016/j.proeng.2015.12.577

Kim YL, Ahn KH, Song JB (2016) “Direct teaching algorithm based on task assistance for machine tending,” 2016 13th International Conference on Ubiquitous Robots and Ambient Intelligence, URAI 2016, pp 861–866.  https://doi.org/10.1109/URAI.2016.7733996

Klimchik A, Ambiehl A, Garnier S, Furet B, Pashkevich A (2016) Experimental study of robotic-based machining. IFAC-PapersOnLine 49(12):174–179. https://doi.org/10.1016/j.ifacol.2016.07.591

Annem V, Rajendran P, Thakar S, Gupta SK (2019) “Towards remote teleoperation of a semi-autonomous mobile manipulator system in machine tending tasks,” ASME 2019 14th International Manufacturing Science and Engineering Conference, MSEC 2019, vol. 1, pp 1–12.  https://doi.org/10.1115/MSEC2019-3027

Zhang J, Fang X (2017) Challenges and key technologies in robotic cell layout design and optimization. Proc Inst Mech Eng C J Mech Eng Sci 231(15):2912–2924. https://doi.org/10.1177/0954406216642473

Jia F, Tzintzun J, Ahmad R (2020) An improved robot path planning algorithm for a novel self-adapting intelligent machine tending robotic system. In: Mechanisms and Machine Science, vol 86. Springer International Publishing, pp 53–64. https://doi.org/10.1007/978-3-030-45402-9_7

Chapter   Google Scholar  

Al-Hussaini S et al (2020) Human-supervised semi-autonomous mobile manipulators for safely and efficiently executing machine tending tasks

Zhong RY, Xu X, Klotz E, Newman ST (2017) Intelligent manufacturing in the context of industry 4.0: a review. Engineering 3(5):616–630. https://doi.org/10.1016/J.ENG.2017.05.015

Page MJ et al (2021) The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ 372. https://doi.org/10.1136/bmj.n71

Sardelis A et al (2020) 2-Stage vision system for robotic handling of flexible objects. Procedia CIRP 97:491–496. https://doi.org/10.1016/j.procir.2020.07.008

Wu X, Li P, Zhou J, Liu Y (2022) A cascaded CNN-based method for monocular vision robotic grasping. Ind Robot 49(4):645–657. https://doi.org/10.1108/IR-10-2021-0236

Moru DK, Borro D (2020) A machine vision algorithm for quality control inspection of gears. Int J Adv Manuf Technol 106(1–2):105–123. https://doi.org/10.1007/s00170-019-04426-2

El Helou M et al (2022) A Modular smart vision system for industrial inspection and control of conformity. Smart Sustain Manuf Syst 6(1):177–189. https://doi.org/10.1520/SSMS20220003

Zhou G et al (2021) “A framework of industrial operations for hybrid robots,” 2021 26th International Conference on Automation and Computing: System Intelligence through Automation and Computing, ICAC 2021, pp 1–6.  https://doi.org/10.23919/ICAC50006.2021.9594267

Jia F, Jebelli A, Ma Y, Ahmad R (2022) An intelligent manufacturing approach based on a novel deep learning method for automatic machine and working status recognition. Appl Sci 12(11):5697. https://doi.org/10.3390/app12115697

Article   CAS   Google Scholar  

Sebbata W, Kenk MA, Brethé JF (2020) “An adaptive robotic grasping with a 2-finger gripper based on deep learning network,” IEEE International Conference on Emerging Technologies and Factory Automation, ETFA, vol. 2020-Septe, pp 620–627.  https://doi.org/10.1109/ETFA46521.2020.9212163

Zheng J, Chen L, Li Y, Khan YA, Lyu H, Wu X (2023) “An intelligent robot sorting system by deep learning on RGB-D image,” 2023 22nd International Symposium INFOTEH-JAHORINA, INFOTEH 2023, no. March, pp 1–8.  https://doi.org/10.1109/INFOTEH57020.2023.10094093

Aliev K, Antonelli D (2019) Analysis of cooperative industrial task execution by mobile and manipulator robots, vol 1. Springer International Publishing. https://doi.org/10.1007/978-3-030-18715-6_21

Book   Google Scholar  

Arrais R et al (2019) “Application of the open scalable production system to machine tending of additive manufacturing operations by a mobile manipulator,” Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 11805 LNAI, pp 345–356.  https://doi.org/10.1007/978-3-030-30244-3_29

Yang R, Nguyen TP, Park SH, Yoon J (2022) Automated picking-sorting system for assembling components in an IKEA chair based on the robotic vision system. Int J Comput Integr Manuf 35(6):583–597. https://doi.org/10.1080/0951192X.2021.1992658

Ma Y, Zhu W, Zhou Y (2022) Automatic grasping control of mobile robot based on monocular vision. Int J Adv Manuf Technol 121(3-4):1785–1798. https://doi.org/10.1007/s00170-022-09438-z

Zhang X, Eltouny K, Liang X, Behdad S (2023) Automatic screw detection and tool recommendation system for robotic disassembly. J Manuf Sci Eng 145(3):1–8. https://doi.org/10.1115/1.4056074

Yang X, Zhou Z, Sørensen JH, Christensen CB, Ünalan M, Zhang X (2023) Automation of SME production with a Cobot system powered by learning-based vision. Robot Comput Integr Manuf 83:102564. https://doi.org/10.1016/j.rcim.2023.102564

Hirz M, Walzel B, Brunner H (2021) Autonomous charging of electric vehicles in industrial environment. Tehnicki Glasnik 15(2):220–225. https://doi.org/10.31803/tg-20210428191147

Transeth AA, Stepanov A, Linnerud AS, Ening K, Gjerstad T (2020) “Competitive high variance, low volume manufacturing with robot manipulators,” 2020 3rd International Symposium on Small-Scale Intelligent Manufacturing Systems, SIMS 2020, vol. m.   https://doi.org/10.1109/SIMS49386.2020.9121464

Guerra-Zubiaga D et al (2022) “Computer vision and machine learning to create an advanced pick-and-place robotic operation using industry 4.0 Trends,” ASME International Mechanical Engineering Congress and Exposition, Proceedings (IMECE), vol. 2-A, pp 1–10.  https://doi.org/10.1115/IMECE2022-89743

Riedlinger MA, Ghafarian Tamizi M, Tikekar J, Redeker M (2022) “Concept for a distributed picking application utilizing robotics and digital twins,” IEEE International Conference on Emerging Technologies and Factory Automation, ETFA, vol. 2022-Septe, pp 1–4.  https://doi.org/10.1109/ETFA52439.2022.9921659

Holst C, Yavuz TB, Gupta P, Ganser P, Bergs T (2022) Deep learning and rule-based image processing pipeline for automated metal cutting tool wear detection and measurement. IFAC-PapersOnLine 55(2):534–539. https://doi.org/10.1016/j.ifacol.2022.04.249

Necula M, Câmpean E, Morar L (2022) Defining the characteristics needed for the cobots design used for the supply of CNC machines. Acta Tech Napocensis Ser Appl Math Mech Eng 65:171–178

Google Scholar  

Zemzemoglu M, Unel M (2022) “Design and implementation of a vision based in-situ defect detection system of automated fiber placement process,” IEEE International Conference on Industrial Informatics (INDIN), vol. 2022-July, pp 393–398.  https://doi.org/10.1109/INDIN51773.2022.9976182

Mao Y, Chen C, Jiang H (2022) “Design and implementation of sorting system based on machine vision,” 2022 7th International Conference on Big Data Analytics, ICBDA 2022, pp 259–264. https://doi.org/10.1109/ICBDA55095.2022.9760324

Ma H, Li L, Shao F, Liu X (2022) Design of a comprehensive experimental platform for intelligent robots based on machine vision, vol. 880 LNEE. Springer Singapore.  https://doi.org/10.1007/978-981-19-0572-8_49

Mei Z, Li D, Chen T, Yu C (2022) “Design of vision system of transceiver robot based on Halcon,” 2022 IEEE International Conference on Artificial Intelligence and Computer Applications, ICAICA 2022, pp 204–207.  https://doi.org/10.1109/ICAICA54878.2022.9844593

D’Souza F, Costa J, Pires JN (2020) Development of a solution for adding a collaborative robot to an industrial AGV. Ind Robot 47(5):723–735. https://doi.org/10.1108/IR-01-2020-0004

Phan HA et al (2021) “Development of a vision system to enhance the reliability of the pick-and-place robot for autonomous testing of camera module used in smartphones,” 7th International Conference on Engineering and Emerging Technologies, ICEET 2021, no. October, pp 1–6. https://doi.org/10.1109/ICEET53442.2021.9659578

Park J, Jun MBG, Yun H (2022) Development of robotic bin picking platform with cluttered objects using human guidance and convolutional neural network (CNN). J Manuf Syst 63:539–549. https://doi.org/10.1016/j.jmsy.2022.05.011

Jadhav T, Jadhav M, Chitre A, Patil A (2022) “Development of vision based test jig and mechanism for automatic inspection and sorting of industrial objects,” 2022 6th International Conference on Computing, Communication, Control and Automation, ICCUBEA 2022, pp 1–5.  https://doi.org/10.1109/ICCUBEA54992.2022.10010756

Wu M, Guo F, Wu J, Xiao Y, Jin M, Zhang Q (2023) “Emergency stop system of computer vision workstation based on GMM-HMM and LSTM,” 2023 9th International Conference on Automation, Robotics and Applications, ICARA 2023, pp 150–154.  https://doi.org/10.1109/ICARA56516.2023.10125926

Jafar FA, Zakaria NA, Noor AZM, Yokota K (2022) “Environmental visual features based place recognition in manufacturing environment,” in Intelligent Manufacturing and Mechatronics. SympoSIMM 2021, vol. 1, M. N. Ali Mokhtar, Z. Jamaludin, M. S. Abdul Aziz, M. N. Maslan, and J. A. Razak, Eds., in Lecture Notes in Mechanical Engineering, vol. 1, Singapore: Springer Nature Singapore, pp 47–59.  https://doi.org/10.1007/978-981-16-8954-3_6

Pradhan AA, Martin WC, Ruiz JD, Deierling PE (2021) “Framework for automated robotic arm manipulation in variable industrial environments,” ASME International Mechanical Engineering Congress and Exposition, Proceedings (IMECE), vol. 2B-2021, pp 1–9.  https://doi.org/10.1115/IMECE2021-71479

Wu K, Cao J (2020) “High robust vision system of robot in intelligent manufacturing,” Proceedings - 2020 Chinese Automation Congress, CAC 2020, no. 2019, pp 4210–4215.  https://doi.org/10.1109/CAC51589.2020.9327823

Lins RG, de Araujo PRM, Corazzim M (2020) In-process machine vision monitoring of tool wear for cyber-physical production systems. Robot Comput Integr Manuf 61:101859. https://doi.org/10.1016/j.rcim.2019.101859

Zhou Z, Li L, Fürsterling A, Durocher HJ, Mouridsen J, Zhang X (2022) Learning-based object detection and localization for a mobile robot manipulator in SME production. Robot Comput Integr Manuf 73:102229. https://doi.org/10.1016/j.rcim.2021.102229

Chen Q, Heydari B, Moghaddam M (2021) Leveraging task modularity in reinforcement learning for adaptable industry 4.0 automation. J Mech Des Trans ASME 143(7). https://doi.org/10.1115/1.4049531

Comari S et al (2022) Mobile cobots for autonomous raw-material feeding of automatic packaging machines. J Manuf Syst 64:211–224. https://doi.org/10.1016/j.jmsy.2022.06.007

Horvath D, Erdos G, Istenes Z, Horvath T, Foldi S (2023) Object Detection using Sim2Real domain randomization for robotic applications. IEEE Trans Robot 39(2):1225–1243. https://doi.org/10.1109/TRO.2022.3207619

Cho J, Kang S, Kim K (2022) Real-time precise object segmentation using a pixel-wise coarse-fine method with deep learning for automated manufacturing. J Manuf Syst 62:114–123. https://doi.org/10.1016/j.jmsy.2021.11.004

Jia F, Ma Y, Ahmad R (2021) Vision-based associative robotic recognition of working status in autonomous manufacturing environment. Procedia CIRP 104:1535–1540. https://doi.org/10.1016/j.procir.2021.11.259

Mamledesai H, Soriano MA, Ahmad R (2020) “A qualitative tool condition monitoring framework using convolution neural network and transfer learning,” Appl Sci, vol. 10, no. 20, p 7298. https://doi.org/10.3390/app10207298

Alcácer V, Cruz-Machado V (2019) Scanning the Industry 4.0: a Literature review on technologies for manufacturing systems. Eng Sci Technol Int J 22(3):899–919. https://doi.org/10.1016/j.jestch.2019.01.006

Bonci A, Cheng PDC, Indri M, Nabissi G, Sibona F (2021) Human-robot perception in industrial environments: a survey. Sensors 21(5):1–29. https://doi.org/10.3390/s21051571

Park YK, Park J, Kim S, Lee D, Lee S, Um J (2023) Development of digital twin for plug-and-produce of a machine tending system through ISO 21919 interface. Int J Comput Integr Manuf 36(10):1449–1464. https://doi.org/10.1080/0951192X.2023.2189306

Matheson E, Minto R, Zampieri EGG, Faccio M, Rosati G (2019) Human-robot collaboration in manufacturing applications: a review. Robotics 8(4):1–25. https://doi.org/10.3390/robotics8040100

Domae Y (2019) Recent trends in the research of industrial robots and future outlook. J Robot Mechatron 31(1):57–62. https://doi.org/10.20965/jrm.2019.p0057

Zeng A et al (2022) Robotic pick-and-place of novel objects in clutter with multi-affordance grasping and cross-domain image matching. Int J Robot Res 41(7):690–705. https://doi.org/10.1177/0278364919868017

Rodriguez-Guerra D, Sorrosal G, Cabanes I, Calleja C (2021) Human-robot interaction review: challenges and solutions for modern industrial environments. IEEE Access 9:108557–108578. https://doi.org/10.1109/ACCESS.2021.3099287

Gualtieri L, Rauch E, Vidoni R (2021) Emerging research fields in safety and ergonomics in industrial collaborative robotics: a systematic literature review. Robot Comput Integr Manuf 67. https://doi.org/10.1016/j.rcim.2020.101998

Demir KA, Döven G, Sezen B (2019) Industry 5.0 and human-robot co-working. Procedia Comput Sci 158:688–695. https://doi.org/10.1016/j.procs.2019.09.104

Coronado E, Kiyokawa T, Ricardez GAG, Ramirez-Alpizar IG, Venture G, Yamanobe N (2022) Evaluating quality in human-robot interaction: a systematic search and classification of performance and human-centered factors, measures and metrics towards an industry 5.0. J Manuf Syst 63:392–410. https://doi.org/10.1016/j.jmsy.2022.04.007

Maddikunta PKR et al (2022) Industry 5.0: a survey on enabling technologies and potential applications. J Ind Inf Integr 26:100257. https://doi.org/10.1016/j.jii.2021.100257

Boøgh S, Hvilshoøj M, Kristiansen M, Madsen O (2012) Identifying and evaluating suitable tasks for autonomous industrial mobile manipulators (AIMM). Int J Adv Manuf Technol. https://doi.org/10.1007/s00170-011-3718-3

Oyekanlu EA et al (2020) A review of recent advances in automated guided vehicle technologies: integration challenges and research areas for 5G-based smart manufacturing applications. IEEE Access 8:202312–202353. https://doi.org/10.1109/ACCESS.2020.3035729

Arvin F, Watson S, Turgut AE, Espinosa J, Krajník T, Lennox B (2018) Perpetual robot swarm: long-term autonomy of mobile robots using on-the-fly inductive charging. J Intell Robot Syst 92(3–4):395–412. https://doi.org/10.1007/s10846-017-0673-8

Vajgl M, Hurtik P, Nejezchleba T (2022) Dist-YOLO: Fast object detection with distance estimation. Appl Sci 12(3):1354. https://doi.org/10.3390/app12031354

Javaid M, Haleem A, Singh RP, Rab S, Suman R (2022) Exploring impact and features of machine vision for progressive industry 4.0 culture. Sens Int 3:100132. https://doi.org/10.1016/j.sintl.2021.100132

Silva RL, Rudek M, Szejka AL, Canciglieri Junior O (2018) Machine vision systems for industrial quality control inspections, vol 540. Springer International Publishing. https://doi.org/10.1007/978-3-030-01614-2_58

Pérez L, Rodríguez Í, Rodríguez N, Usamentiaga R, García DF (2016) Robot guidance using machine vision techniques in industrial environments: a comparative review. Sensors 16(3). https://doi.org/10.3390/s16030335

Awcock GJ, Thomas R (1995) Applied image processing. Macmillan Education UK, London. https://doi.org/10.1007/978-1-349-13049-8

Zou Z, Chen K, Shi Z, Guo Y, Ye J (2023) Object detection in 20 years: a survey. Proceedings of the IEEE 111(3):257–276. https://doi.org/10.1109/JPROC.2023.3238524

Nguyen TTH, Jatowt A, Coustaty M, Doucet A (2021) Survey of post-OCR processing approaches. ACM Comput Surv 54(6). https://doi.org/10.1145/3453476

Zhao QJ, Cao P, Tu DW (2014) Toward intelligent manufacturing: label characters marking and recognition method for steel products with machine vision. Adv Manuf 2(1):3–12. https://doi.org/10.1007/s40436-014-0057-2

Shahin M, Chen FF, Hosseinzadeh A (2023) Machine-based identification system via optical character recognition. Flex Serv Manuf J (0123456789). https://doi.org/10.1007/s10696-023-09497-8

Koponen J, Haataja K, Toivanen P (2023) A novel deep learning method for recognizing texts printed with multiple different printing methods. F1000Research 12:427. https://doi.org/10.12688/f1000research.131775.1

Caldeira T, Ciarelli PM, Neto GA (2020) Industrial optical character recognition system in printing quality control of hot-rolled coils identification. J Control Autom Electr Syst 31(1):108–118. https://doi.org/10.1007/s40313-019-00551-1

Zeba G, Dabić M, Čičak M, Daim T, Yalcin H (2021) Technology mining: artificial intelligence in manufacturing. Technol Forecast Soc Change 171. https://doi.org/10.1016/j.techfore.2021.120971

Wuest T, Weimer D, Irgens C, Thoben KD (2016) Machine learning in manufacturing: advantages, challenges, and applications. Prod Manuf Res 4(1):23–45. https://doi.org/10.1080/21693277.2016.1192517

Melhem M, Ananou B, Ouladsine M, Pinaton J (2016) Regression methods for predicting the product’s quality in the semiconductor manufacturing process. IFAC-PapersOnLine 49(12):83–88. https://doi.org/10.1016/j.ifacol.2016.07.554

Papananias M, McLeay TE, Mahfouf M, Kadirkamanathan V (2019) A Bayesian framework to estimate part quality and associated uncertainties in multistage manufacturing. Comput Ind 105:35–47. https://doi.org/10.1016/j.compind.2018.10.008

Akdulum A, Kayir Y (2023) “Prediction of thrust force in indexable drilling of aluminum alloys with machine learning algorithms,” Measurement, vol. 222. https://doi.org/10.1016/j.measurement.2023.113655

Jiang C, Wan J, Abbas H (2021) An edge computing node deployment method based on improved k-means clustering algorithm for smart manufacturing. IEEE Syst J 15(2):2230–2240. https://doi.org/10.1109/JSYST.2020.2986649

Article   ADS   Google Scholar  

Akdulum A, Kayir Y “Modeling and estimation of thrust force, torque, and surface roughness in indexable drilling of AA6061-T651 with Taguchi, ANN, and ANFIS”.  https://doi.org/10.1007/s12046-023-02209-wS

Lecun Y, Bengio Y, Hinton G (2015) Deep learning. Nature 521(7553):436–444. https://doi.org/10.1038/nature14539

Article   ADS   CAS   PubMed   Google Scholar  

Xu R, Yan W (2020) “A comparison of GANs-based approaches for combustor system fault detection,” Proceedings of the International Joint Conference on Neural Networks.  https://doi.org/10.1109/IJCNN48605.2020.9207487

Singh R, Garg R, Patel NS, Braun MW (2020) “Generative adversarial networks for synthetic defect generation in assembly and test manufacturing,” ASMC (Advanced Semiconductor Manufacturing Conference) Proceedings, vol. 2020-Augus, pp 1–5.  https://doi.org/10.1109/ASMC49169.2020.9185242

Zhou L, Zhang L, Konz N (2023) Computer vision techniques in manufacturing. IEEE Trans Syst Man Cybern Syst 53(1):105–117. https://doi.org/10.1109/TSMC.2022.3166397

Huang SH et al (2019) Learning gentle object manipulation with curiosity-driven deep reinforcement learning

Wang J, Ma Y, Zhang L, Gao RX, Wu D (2018) Deep learning for smart manufacturing: methods and applications. J Manuf Syst 48:144–156. https://doi.org/10.1016/j.jmsy.2018.01.003

Wu C, Du H, Wu Q, Zhang S (2020) Image text deblurring method based on generative adversarial network. Electronics 9(2):1–14. https://doi.org/10.3390/electronics9020220

Amanlou A, Suratgar AA, Tavoosi J, Mohammadzadeh A, Mosavi A (2022) Single-image reflection removal using deep learning: a systematic review. IEEE Access 10:29937–29953. https://doi.org/10.1109/ACCESS.2022.3156273

Lange D (2019) Cognitive robotics: making robots sense, understand, and interact. Computer 52(12):39–44. https://doi.org/10.1109/MC.2019.2942579

Salman AE, Roman MR (2023) “Augmented reality-assisted gesture-based teleoperated system for robot motion planning,” Ind Robot, vol. 50, no. 5, pp 765–780.  https://doi.org/10.1108/IR-11-2022-0289

Jeong R et al (2020) “Learning dexterous manipulation from suboptimal experts,” pp 1–20

Jang S (2023) “MURM: utilization of multi-views for goal-conditioned reinforcement learning in robotic manipulation”

Bergmann P, Fauser M, Sattlegger D, Steger C (2019) “MVTEC ad-A comprehensive real-world dataset for unsupervised anomaly detection,” Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, vol. 2019-June, pp 9584–9592. https://doi.org/10.1109/CVPR.2019.00982

Chen T, Kornblith S, Norouzi M, Hinton G (2020) “A simple framework for contrastive learning of visual representations,” 37th International Conference on Machine Learning, ICML 2020, vol. PartF16814, no. Figure 1, pp 1575–1585

Download references

Acknowledgements

We express our gratitude to the Ministry of Economic Development, Trade, and Tourism of the Government of Alberta for funding this project through Autonomous Systems Initiative of the Major Innovation Funds, and the Go Productivity funding. The authors also would like to acknowledge the NSERC (Grant Nos. NSERC RGPIN-2017-04516 and NSERC CRDPJ 537378-18) for further funding this project.

Author information

Authors and affiliations.

Smart & Sustainable Manufacturing Systems Laboratory (SMART LAB), Department of Mechanical Engineering, University of Alberta, Edmonton, Canada

Feiyu Jia & Rafiq Ahmad

Department of Mechanical and Energy Engineering, Southern University of Science and Technology, Shenzhen, China

Yongsheng Ma

You can also search for this author in PubMed   Google Scholar

Contributions

Feiyu Jia: conceptualization, investigation, methodology, software, validation, writing – original draft. Yongsheng Ma: methodology, writing – review & editing. Rafiq Ahmad: conceptualization, supervision, methodology, writing – review & editing, project administration, funding acquisition.

Corresponding author

Correspondence to Rafiq Ahmad .

Ethics declarations

Competing interests.

The authors declare no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Jia, F., Ma, Y. & Ahmad, R. Review of current vision-based robotic machine-tending applications. Int J Adv Manuf Technol 131 , 1039–1057 (2024). https://doi.org/10.1007/s00170-024-13168-9

Download citation

Received : 07 September 2023

Accepted : 30 January 2024

Published : 15 February 2024

Issue Date : March 2024

DOI : https://doi.org/10.1007/s00170-024-13168-9

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Manufacturing
• Machine tending
• Computer vision
• artificial intelligence
• Deep learning
• Find a journal
• Publish with us
• Track your research

Machine Vision in the Context of Robotics: A Systematic Literature Review

Machine vision is critical to robotics due to a wide range of applications which rely on input from visual sensors such as autonomous mobile robots and smart production systems. To create the smart homes and systems of tomorrow, an overview about current challenges in the research field would be of use to identify further possible directions, created in a systematic and reproducible manner. In this work a systematic literature review was conducted covering research from the last 10 years. We screened 172 papers from four databases and selected 52 relevant papers. While robustness and computation time were improved greatly, occlusion and lighting variance are still the biggest problems faced. From the number of recent publications, we conclude that the observed field is of relevance and interest to the research community. Further challenges arise in many areas of the field.

Javad Ghofrani

Robert Kirschne

Daniel Rossburg

Dirk Reichelt

Related Research

The security of smart buildings: a systematic literature review, how to certify machine learning based safety-critical systems a systematic literature review, distilling information from a flood: a possibility for the use of meta-analysis and systematic review in machine learning research, state-of-the-art in smart contact lenses for human machine interaction, embodiment and computational creativity, a decade of information architecture in hci: a systematic literature review, nanorobotics in medicine: a systematic review of advances, challenges, and future prospects.

Please sign up or login with your details

Generation Overview

AI Generator calls

AI Video Generator calls

AI Chat messages

Genius Mode messages

Genius Mode images

AD-free experience

Private images

• Includes 500 AI Image generations, 1750 AI Chat Messages, 30 AI Video generations, 60 Genius Mode Messages and 60 Genius Mode Images per month. If you go over any of these limits, you will be charged an extra $5 for that group.
• For example: if you go over 500 AI images, but stay within the limits for AI Chat and Genius Mode, you'll be charged $5 per additional 500 AI Image generations.
• Includes 100 AI Image generations and 300 AI Chat Messages. If you go over any of these limits, you will have to pay as you go.
• For example: if you go over 100 AI images, but stay within the limits for AI Chat, you'll have to reload on credits to generate more images. Choose from $5 - $1000. You'll only pay for what you use.

Out of credits

Refill your membership to continue using DeepAI

Share your generations with friends

• DOI: 10.1051/E3SCONF/202123604027
• Corpus ID: 234242524

Literature Review of Machine Vision in Application Field

• Chen Zhang , Xuewu Xu , +1 author Guoping Wang
• Published 2021
• Computer Science, Engineering

6 Citations

Computer vision based quality control for additive manufacturing parts, a deep dive into robot vision - an integrative systematic literature review methodologies and research endeavor practices, errors in measuring the distance to an obstacle by technical vision means and in forecasting braking distance in driverless train control systems. world of transport and transportation, exploring the vididetect tool for automated defect detection in manufacturing with machine vision, an automatic defect detection system for petrochemical pipeline based on cycle-gan and yolo v5, peak density algorithm based on kd-tree optimization, 3 references, using open source libraries in the development of control systems based on machine vision, an optoneuronic device with realistic retinal expressions for bioinspired machine vision, workspace supervising system for material handling devices with machine vision assistance, related papers.

Showing 1 through 3 of 0 Related Papers

•  Sign into My Research
•  Create My Research Account
• Company Website
• Our Products
• About Dissertations
• Español (España)
• Support Center

Find your institution

Examples: State University, [email protected]

Other access options

• More options

Select language

• Bahasa Indonesia
• Português (Brasil)
• Português (Portugal)

Welcome to My Research!

You may have access to the free features available through My Research. You can save searches, save documents, create alerts and more. Please log in through your library or institution to check if you have access.

Translate this article into 20 different languages!

If you log in through your library or institution you might have access to this article in multiple languages.

Get access to 20+ different citations styles

Styles include MLA, APA, Chicago and many more. This feature may be available for free if you log in through your library or institution.

Looking for a PDF of this document?

You may have access to it for free by logging in through your library or institution.

Want to save this document?

You may have access to different export options including Google Drive and Microsoft OneDrive and citation management tools like RefWorks and EasyBib. Try logging in through your library or institution to get access to these tools.

• Document 1 of 1
• More like this
• Conference Paper

Literature Review of Machine Vision in Application Field

No items selected.

Please select one or more items.

Select results items first to use the cite, email, save, and export options

[[missing key: loading-pdf-error]] [[missing key: loading-pdf-link]]

Aiming at the application and research of machine vision, a comprehensive and detailed elaboration is carried out in its two application areas: visual inspection and robot vision. Introduce the composition, characteristics and application advantages of the machine vision system. Based on the analysis of the current research status at home and abroad, the application development trend of machine vision is prospected.

You have requested "on-the-fly" machine translation of selected content from our databases. This functionality is provided solely for your convenience and is in no way intended to replace human translation. Show full disclaimer

Neither ProQuest nor its licensors make any representations or warranties with respect to the translations. The translations are automatically generated "AS IS" and "AS AVAILABLE" and are not retained in our systems. PROQUEST AND ITS LICENSORS SPECIFICALLY DISCLAIM ANY AND ALL EXPRESS OR IMPLIED WARRANTIES, INCLUDING WITHOUT LIMITATION, ANY WARRANTIES FOR AVAILABILITY, ACCURACY, TIMELINESS, COMPLETENESS, NON-INFRINGMENT, MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Your use of the translations is subject to all use restrictions contained in your Electronic Products License Agreement and by using the translation functionality you agree to forgo any and all claims against ProQuest or its licensors for your use of the translation functionality and any output derived there from. Hide full disclaimer

Suggested sources

• About ProQuest
• Terms of Use
• Privacy Policy
• Cookie Policy

• Research article
• Open access
• Published: 15 February 2021

Systematic literature review of machine learning methods used in the analysis of real-world data for patient-provider decision making

• Alan Brnabic 1 &
• Lisa M. Hess   ORCID: orcid.org/0000-0003-3631-3941 2  

BMC Medical Informatics and Decision Making volume  21 , Article number:  54 ( 2021 ) Cite this article

31k Accesses

56 Citations

3 Altmetric

Metrics details

Machine learning is a broad term encompassing a number of methods that allow the investigator to learn from the data. These methods may permit large real-world databases to be more rapidly translated to applications to inform patient-provider decision making.

This systematic literature review was conducted to identify published observational research of employed machine learning to inform decision making at the patient-provider level. The search strategy was implemented and studies meeting eligibility criteria were evaluated by two independent reviewers. Relevant data related to study design, statistical methods and strengths and limitations were identified; study quality was assessed using a modified version of the Luo checklist.

A total of 34 publications from January 2014 to September 2020 were identified and evaluated for this review. There were diverse methods, statistical packages and approaches used across identified studies. The most common methods included decision tree and random forest approaches. Most studies applied internal validation but only two conducted external validation. Most studies utilized one algorithm, and only eight studies applied multiple machine learning algorithms to the data. Seven items on the Luo checklist failed to be met by more than 50% of published studies.

Conclusions

A wide variety of approaches, algorithms, statistical software, and validation strategies were employed in the application of machine learning methods to inform patient-provider decision making. There is a need to ensure that multiple machine learning approaches are used, the model selection strategy is clearly defined, and both internal and external validation are necessary to be sure that decisions for patient care are being made with the highest quality evidence. Future work should routinely employ ensemble methods incorporating multiple machine learning algorithms.

Peer Review reports

Traditional methods of analyzing large real-world databases (big data) and other observational studies are focused on the outcomes that can inform at the population-based level. The findings from real-world studies are relevant to populations as a whole, but the ability to predict or provide meaningful evidence at the patient level is much less well established due to the complexity with which clinical decision making is made and the variety of factors taken into account by the health care provider [ 1 , 2 ]. Using traditional methods that produce population estimates and measures of variability, it is very challenging to accurately predict how any one patient will perform, even when applying findings from subgroup analyses. The care of patients is nuanced, and multiple non-linear, interconnected factors must be taken into account in decision making. When data are available that are only relevant at the population level, health care decision making is less informed as to the optimal course of care for a given patient.

Clinical prediction models are an approach to utilizing patient-level evidence to help inform healthcare decision makers about patient care. These models are also known as prediction rules or prognostic models and have been used for decades by health care professionals [ 3 ]. Traditionally, these models combine patient demographic, clinical and treatment characteristics in the form of a statistical or mathematical model, usually regression, classification or neural networks, but deal with a limited number of predictor variables (usually below 25). The Framingham Heart Study is a classic example of the use of longitudinal data to build a traditional decision-making model. Multiple risk calculators and estimators have been built to predict a patient’s risk of a variety of cardiovascular outcomes, such as atrial fibrillation and coronary heart disease [ 4 , 5 , 6 ]. In general, these studies use multivariable regression evaluating risk factors identified in the literature. Based on these findings, a scoring system is derived for each factor to predict the likelihood of an adverse outcome based on a patient’s score across all risk factors evaluated.

With the advent of more complex data collection and readily available data sets for patients in routine clinical care, both sample sizes and potential predictor variables (such as genomic data) can exceed the tens of thousands, thus establishing the need for alternative approaches to rapidly process a large amount of information. Artificial intelligence (AI), particularly machine learning methods (a subset of AI), are increasingly being utilized in clinical research for prediction models, pattern recognition and deep-learning techniques used to combine complex information for example genomic and clinical data [ 7 , 8 , 9 ]. In the health care sciences, these methods are applied to replace a human expert to perform tasks that would otherwise take considerable time and expertise, and likely result in potential error. The underlying concept is that a machine will learn by trial and error from the data itself, to make predictions without having a pre-defined set of rules for decision making. Simply, machine learning can simply be better understood as “learning from data.” [ 8 ].

There are two types of learning from the data, unsupervised and supervised. Unsupervised learning is a type of machine learning algorithm used to draw inferences from datasets consisting of input data without labelled responses. The most common unsupervised learning method is cluster analysis, which is used for exploratory data analysis to find hidden patterns or grouping in data. Supervised learning involves making a prediction based on a set of pre-specified input and output variables. There are a number of statistical tools used for supervised learning. Some examples include traditional statistical prediction methods like regression models (e.g. regression splines, projection pursuit regression, penalized regression) that involve fitting a model to data, evaluating the fit and estimating parameters that are later used in a predictive equation. Other tools include tree-based methods (e.g. classification and regression trees [CART] and random forests), which successively partition a data set based on the relationships between predictor variables and a target (outcome) variable. Other examples include neural networks, discriminant functions and linear classifiers, support vector classifiers and machines. Often, predictive tools are built using various forms of model aggregation (or ensemble learning) that may combine models based on resampled or re-weighted data sets. These different types of models can be fitted to the same data using model averaging.

Classical statistical regression methods used for prediction modeling are well understood in the statistical sciences and the scientific community that employs them. These methods tend to be transparent and are usually hypothesis driven but can overlook complex associations with limited flexibility when a high number of variables are investigated. In addition, when using classic regression modeling, choosing the ‘right’ model is not straightforward. Non-traditional machine learning algorithms, and machine learning approaches, may overcome some of these limitations of classical regression models in this new era of big data, but are not a complete solution as they must be considered in the context of the limitations of data used in the analysis [ 2 ].

While machine learning methods can be used for both population-based models as well as for informed patient-provider decision making, it is important to note that the data, model, and outputs used to inform the care of an individual patient must meet the highest standards of research quality, as the choice made will likely have an impact on both the long- and short-term patient outcomes. While a range of uncertainty can be expected for population-based estimates, the risk of error for patient level models must be minimized to ensure quality patient care. The risks and concerns of utilizing machine learning for individual patient decision making have been raised by ethicists [ 10 ]. The risks are not limited to the lack of transparency, limited data regarding the confidence of the findings, and the risk of reducing patient autonomy in choice by relying on data that may foster a more paternalistic model of healthcare. These are all important and valid concerns, and therefore the role of machine learning for patient care must meet the highest standards to ensure that shared, not simply informed, evidence-based decision making be supported by these methods.

A systematic literature review was published in 2018 that evaluated the statistical methods that have been used to enable large, real-world databases to be used at the patient-provider level [ 11 ]. Briefly, this study identified a total of 115 articles that evaluated the use of logistic regression (n = 52, 45.2%), Cox regression (n = 24, 20.9%), and linear regression (n = 17, 14.8%). However, an interesting observation noted several studies utilizing novel statistical approaches such as machine learning, recursive partitioning, and development of mathematical algorithms to predict patient outcomes. More recently, publications are emerging describing the use of Individualized Treatment Recommendation algorithms and Outcome Weighted Learning for personalized medicine using large observational databases [ 12 , 13 ]. Therefore, this systematic literature review was designed to further pursue this observation to more comprehensively evaluate the use of machine learning methods to support patient-provider decision making, and to critically evaluate the strengths and weaknesses of these methods. For the purposes of this work, data supporting patient-provider decision making was defined as that which provided information specifically on a treatment or intervention choice; while both population-based and risk estimator data are certainly valuable for patient care and decision making, this study was designed to evaluate data that would specifically inform a choice for the patient with the provider. The overarching goal is to provide evidence of how large datasets can be used to inform decisions at the patient level using machine learning-based methods, and to evaluate the quality of such work to support informed decision making.

This study originated from a systematic literature review that was conducted in MEDLINE and PsychInfo; a refreshed search was conducted in September 2020 to obtain newer publications (Table 1 ). Eligible studies were those that analyzed prospective or retrospective observational data, reported quantitative results, and described statistical methods specifically applicable to patient-level decision making. Specifically, patient-level decision making referred to studies that provided data for or against a particular intervention at the patient level, so that the data could be used to inform decision making at the patient-provider level. Studies did not meet this criterion if only a population-based estimates, mortality risk predictors, or satisfaction with care were evaluated. Additionally, studies designed to improve diagnostic tools and those evaluating health care system quality indicators did not meet the patient-provider decision-making criterion. Eligible statistical methods for this study were limited to machine learning-based approaches. Eligibility was assessed by two reviewers and any discrepancies were discussed; a third reviewer was available to serve as a tie breaker in case of different opinions. The final set of eligible publications were then abstracted into a Microsoft Excel document. Study quality was evaluated using a modified Luo scale, which was developed specifically as a tool to standardize high-quality publication of machine learning models [ 14 ]. A modified version of this tool was utilized for this study; specifically, the optional item were removed, and three terms were clarified: item 6 (define the prediction problem) was redefined as “define the model,” item 7 (prepare data for model building) was renamed “model building and validation,” and item 8 (build the predictive model) was renamed “model selection” to more succinctly state what was being evaluated under each criterion. Data were abstracted and both extracted data and the Luo checklist items were reviewed and verified by a second reviewer to ensure data comprehensiveness and quality. In all cases of differences in eligibility assessment or data entry, the reviewers met and ensured agreement with the final set of data to be included in the database for data synthesis, with a third reviewer utilized as a tie breaker in case of discrepancies. Data were summarized descriptively and qualitatively, based on the following categories: publication and study characteristics; patient characteristics; statistical methodologies used, including statistical software packages; strengths and weaknesses; and interpretation of findings.

The search strategy was run on September 1, 2020 and identified a total of 34 publications that utilized machine learning methods for individual patient-level decision making (Fig.  1 ). The most common reason for study exclusion, as expected, was due to the study not meeting the patient-level decision making criterion. A summary of the characteristics of eligible studies and the patient data are included in Table 2 . Most of the real-world data sources included retrospective databases or designs (n = 27, 79.4%), primarily utilizing electronic health records. Six analyses utilized prospective cohort studies and one utilized data from a cross sectional study.

Prisma diagram of screening and study identification

General approaches to machine learning

The types of classification or prediction machine learning algorithms are reported in Table 2 . These included decision tree/random forest analyses (19 studies) [ 15 , 16 , 17 , 18 , 19 , 20 , 21 , 22 , 23 , 24 , 25 , 26 , 27 , 28 , 29 , 30 , 31 , 32 , 33 ] and neural networks (19 studies) [ 24 , 25 , 26 , 27 , 28 , 29 , 30 , 32 , 34 , 35 , 36 , 37 , 38 , 39 , 40 , 41 , 42 , 43 , 44 ]. Other approaches included latent growth mixture modeling [ 45 ], support vector machine classifiers [ 46 ], LASSO regression [ 47 ], boosting methods [ 23 ], and a novel Bayesian approach [ 26 , 40 , 48 ]. Within the analytical approaches to support machine learning, a variety of methods were used to evaluate model fit, such as Akaike Information Criterion, Bayesian Information Criterion, and the Lo-Mendel-Rubin likelihood ratio test [ 22 , 45 , 47 ], and while most studies included the area under the curve (AUC) of receiver-operator characteristic (ROC) curves (Table 3 ), analyses also included sensitivity/specificity [ 16 , 19 , 24 , 30 , 41 , 42 , 43 ], positive predictive value [ 21 , 26 , 32 , 38 , 40 , 41 , 42 , 43 ], and a variety of less common approaches such as the geometric mean [ 16 ], use of the Matthews correlation coefficient (ranges from -1.0, completely erroneous information, to + 1.0, perfect prediction) [ 46 ], defining true/false negatives/positives by means of a confusion matrix [ 17 ], calculating the root mean square error of the predicted versus original outcome profiles [ 37 ], or identifying the model with the best average performance training and performance cross validation [ 36 ].

Statistical software packages

The statistical programs used to perform machine learning varied widely across these studies, no consistencies were observed (Table 2 ). As noted above, one study using decision tree analysis used Quinlan’s C5.0 decision tree algorithm [ 15 ] while a second used an earlier version of this program (C4.5) [ 20 ]. Other decision tree analyses utilized various versions of R [ 18 , 19 , 22 , 24 , 27 , 47 ], International Business Machines (IBM) Statistical Package for the Social Sciences (SPSS) [ 16 , 17 , 33 , 47 ], the Azure Machine Learning Platform [ 30 ], or programmed the model using Python [ 23 , 25 , 46 ]. Artificial neural network analyses used Neural Designer [ 34 ] or Statistica V10 [ 35 ]. Six studies did not report the software used for analysis [ 21 , 31 , 32 , 37 , 41 , 42 ].

Families of machine learning algorithms

Also as summarized in Table 2 , more than one third of all publications (n = 13, 38.2%) applied only one family of machine learning algorithm to model development [ 16 , 17 , 18 , 19 , 20 , 34 , 37 , 41 , 42 , 43 , 46 , 48 ]; and only four studies utilized five or more methods [ 23 , 25 , 28 , 45 ]. One applied an ensemble of six different algorithms and the software was set to run 200 iterations [ 23 ], and another ran seven algorithms [ 45 ].

Internal and external validation

Evaluation of study publication quality identified the most common gap in publications as the lack of external validation, which was conducted by only two studies [ 15 , 20 ]. Seven studies predefined the success criteria for model performance [ 20 , 21 , 23 , 35 , 36 , 46 , 47 ], and five studies discussed the generalizability of the model [ 20 , 23 , 34 , 45 , 48 ]. Six studies [ 17 , 18 , 21 , 22 , 35 , 36 ] discussed the balance between model accuracy and model simplicity or interpretability, which was also a criterion of quality publication in the Luo scale [ 14 ]. The items on the checklist that were least frequently met are presented in Fig.  2 . The complete quality assessment evaluation for each item in the checklist is included in Additional file 1 : Table S1.

Least frequently met study quality items, modified Luo Scale [ 14 ]

There were a variety of approaches taken to validate the models developed (Table 3 ). Internal validation with splitting into a testing and validation dataset was performed in all studies. The cohort splitting approach was conducted in multiple ways, using a 2:1 split [ 26 ], 60/40 split [ 21 , 36 ], a 70/30 split [ 16 , 17 , 22 , 30 , 33 , 35 ], 75/25 split [ 27 , 40 ], 80/20 split [ 46 ], 90/10 split [ 25 , 29 ], splitting the data based on site of care [ 48 ], a 2/1/1 split for training, testing and validation [ 38 ], and splitting 60/20/20, where the third group was selected for model selection purposes prior to validation [ 34 ]. Nine studies did not specifically mention the form of splitting approach used [ 15 , 18 , 19 , 20 , 24 , 29 , 39 , 45 , 47 ], but most of those noted the use of k fold cross validation. One training set corresponded to 90% of the sample [ 23 ], whereas a second study was less clear, as input data were at the observation level with multiple observations per patient, and 3 of the 15 patients were included in the training set [ 37 ]. The remaining studies did not specifically state splitting the data into testing and validation samples, but most specified they performed five-fold cross validation (including one that generally mentioned cohort splitting) [ 18 , 45 ] or ten-fold cross validation strategies [ 15 , 19 , 20 , 28 ].

External validation was conducted by only two studies (5.9%). Hische and colleagues conducted a decision tree analysis, which was designed to identify patients with impaired fasting glucose [ 20 ]. Their model was developed in a cohort study of patients from the Berlin Potsdam Cohort Study (n = 1527) and was found to have a positive predictive value of 56.2% and a negative predictive value of 89.1%. The model was then tested on an independent from the Dresden Cohort (n = 1998) with a family history of type II diabetes. In external validation, positive predictive value was 43.9% and negative predictive value was 90.4% [ 20 ]. Toussi and colleagues conducted both internal and external validation in their decision tree analysis to evaluate individual physician prescribing behaviors using a database of 463 patient electronic medical records [ 15 ]. For the internal validation step, the cross-validation option was used from Quinlan’s C5.0 decision tree learning algorithm as their study sample was too small to split into a testing and validation sample, and external validation was conducted by comparing outcomes to published treatment guidelines. Unfortunately, they found little concordance between physician behavior and guidelines potentially due to the timing of the data not matching the time period in which guidelines were implemented, emphasizing the need for a contemporaneous external control [ 15 ].

Handling of missing values

Missing values were addressed in most studies (n = 21, 61.8%) in this review, but there were thirteen remaining studies that did not mention if there were missing data or how they were handled (Table 3 ). For those that reported methods related to missing data, there were a wide variety of approaches used in real-world datasets. The full information maximum likelihood method was used for estimating model parameters in the presence of missing data for the development of the model by Hertroijs and colleagues, but patients with missing covariate values at baseline were excluded from the validation of the model [ 45 ]. Missing covariate values were included in models as a discrete category [ 48 ]. Four studies removed patients from the model with missing data [ 46 ], resulting in the loss of 16%-41% of samples in three studies [ 17 , 36 , 47 ]. Missing data from primary outcome variables were reported among with 59% (men) and 70% (women) within a study of diabetes [ 16 ]. In this study, single imputation was used; for continuous variables CART (IBM SPSS modeler V14.2.03) and for categorical variables the authors used the weighted K-Nearest Neighbor approach using RapidMiner (V.5) [ 16 ]. Other studies reported exclusion but not specifically the impact on sample size [ 29 , 31 , 38 , 44 ]. Imputation was conducted in a variety of ways for studies with missing data [ 22 , 25 , 28 , 33 ]. Single imputation was used in the study by Bannister and colleagues, but followed by multiple imputation in the final model to evaluate differences in model parameters [ 22 ]. One study imputed with a standard last-imputation-forward approach [ 26 ]. Spline techniques were used to impute missing data in the training set of one study [ 37 ]. Missingness was largely retained as an informative variable, and only variables missing for 85% or more of participants were excluded by Alaa et al. [ 23 ] while Hearn et al. used a combination of imputation and exclusion strategies [ 40 ]. Lastly, missing or incomplete data were imputed using a model-based approach by Toussi et al. [ 15 ] and using an optimal-impute algorithm by Bertsimas et al. [ 21 ].

Strengths and weaknesses noted by authors

Publications summarized the strengths and weaknesses of the machine learning methods employed. Low complexity and simplicity of machine-based learning models were noted as strengths of this approach [ 15 , 20 ]. Machine learning approaches were both powerful and efficient methods to apply to large datasets [ 19 ]. It was noted that parameters in this study that were significant at the patient level were included, even if at the broader population-based level using traditional regression analysis model development they would have not been significant and therefore would have been otherwise excluded using traditional approaches [ 34 ]. One publication noted the value of machine learning being highly dependent on the model selection strategy and parameter optimization, and that machine learning in and of itself will not provide better estimates unless these steps are conducted properly [ 23 ].

Even when properly planned, machine learning approaches are not without issues that deserve attention in future studies that employ these techniques. Within the eligible publications, weaknesses included overfitting the model with the inclusion of too much detail [ 15 ]. Additional limitations are based on the data sources used for machine learning, such as the lack of availability of all desired variables and missing data that can affect the development and performance of these models [ 16 , 34 , 36 , 48 ]. The lack of all relevant variables was noted as a particular concern for retrospective database studies, where the investigator is limited to what has been recorded [ 26 , 28 , 29 , 38 , 40 ]. Importantly and as observed in the studies included in this review, the lack of external validation was stated as a limitation of studies included in this review [ 28 , 30 , 38 , 42 ].

Limitations can also be on the part of the research team, as the need for both clinical and statistical expertise in the development and execution of studies using machine learning-based methodology, and users are warned against applying these methods blindly [ 22 ]. The importance of the role of clinical and statistical experts in the research team was noted in one study and highlighted as a strength of their work [ 21 ].

This study systematically reviewed and summarized the methods and approaches used for machine learning as applied to observational datasets that can inform patient-provider decision making. Machine learning methods have been applied much more broadly across observational studies than in the context of individual decision making, so the summary of this work does not necessarily apply to all machine learning-based studies. The focus of this work is on an area that remains largely unexplored, which is how to use large datasets in a manner that can inform and improve patient care in a way that supports shared decision making with reliable evidence that is applicable to the individual patient. Multiple publications cite the limitations of using population-based estimates for individual decisions [ 49 , 50 , 51 ]. Specifically, a summary statistic at the population level does not apply to each person in that cohort. Population estimates represent a point on a potentially wide distribution, and any one patient could fall anywhere within that distribution and be far from the point estimate value. On the other extreme, case reports or case series provide very specific individual-level data, but are not generalizable to other patients [ 52 ]. This review and summary provides guidance and suggestions of best practices to improve and hopefully increase the use of these methods to provide data and models to inform patient-provider decision making.

It was common for single modeling strategies to be employed within the identified publications. It has long been known that single algorithms to estimation can produce a fair amount of uncertainty and variability [ 53 ]. To overcome this limitation, there is a need for multiple algorithms and multiple iterations of the models to be performed. This, combined with more powerful analytics in recent years, provides a new standard for machine learning algorithm choice and development. While in some cases, a single model may fit the data well and provide an accurate answer, the certainty of the model can be supported through novel approaches, such as model averaging [ 54 ]. Few studies in this review combined multiple families of modeling strategies along with multiple iterations of the models. This should become a best practice in the future and is recommended as an additional criterion to assess study quality among machine learning-based modeling [ 54 ].

External validation is critical to ensure model accuracy, but was rarely conducted in the publications included in this review. The reasons for this could be many, such as lack of appropriate datasets or due to the lack of awareness of the importance of external validation [ 55 ]. As model development using machine learning increases, there is a need for external validation prior to application of models in any patient-provider setting. The generalizability of models is largely unknown without these data. Publications that did not conduct external validation also did not note the need for this to be completed, as generalizability was discussed in only five studies, one of which had also conducted the external validation. Of the remaining four studies, the role of generalizability was noted in terms of the need for future external validation in only one study [ 48 ]. Other reviews that were more broadly conducted to evaluate machine learning methods similarly found a low rate of external validation (6.6% versus 5.9% in this study) [ 56 ]. It was shown that there was lower prediction accuracy by external validation than simply by cross validation alone. The current review, with a focus on machine learning to support decision making at a practical level, suggests external validation is an important gap that should be filled prior to using these models for patient-provider decision making.

Luo and others suggest that k -fold validation may be used with proper stratification of the response variable as part of the model selection strategy [ 14 , 55 ]. The studies identified in this review generally conducted 5- or tenfold validation. There is no formal rule for the selection for the value of k , which is typically based on the size of the dataset; as k increases, bias will be reduced, but in turn variance will increase. While the tradeoff has to be accounted for, k  = 5–10 has been found to be reasonable for most study purposes [ 57 ].

The evidence from identified publications suggests that the ethical concerns of lack of transparency and failure to report confidence in the findings are largely warranted. These limitations can be addressed through the use of multiple modeling approaches (to clarify the ‘black box’ nature of these approaches) and by including both external and high k-fold validation (to demonstrate the confidence in findings). To ensure these methods are used in a manner that improves patient care, the expectations of population-based risk prediction models of the past are no longer sufficient. It is essential that the right data, the right set of models, and appropriate validation are employed to ensure that the resulting data meet standards for high quality patient care.

This study did not evaluate the quality of the underlying real-world data used to develop, test or validate the algorithms. While not directly part of the evaluation in this review, researchers should be aware that all limitations of real-world data sources apply regardless of the methodology employed. However, when observational datasets are used for machine learning-based research, the investigator should be aware of the extent to which the methods they are using depend on the data structure and availability, and should evaluate a proposed data source to ensure it is appropriate for the machine learning project [ 45 ]. Importantly, databases should be evaluated to fully understand the variables included, as well as those variables that may have prognostic or predictive value, but may not be included in the dataset. The lack of important variables remains a concern with the use of retrospective databases for machine learning. The concerns with confounding (particularly unmeasured confounding), bias (including immortal time bias), and patient selection criteria to be in the database must also be evaluated [ 58 , 59 ]. These are factors that should be considered prior to implementing these methods, and not always at the forefront of consideration when applying machine learning approaches. The Luo checklist is a valuable tool to ensure that any machine-learning study meets high research standards for patient care, and importantly includes the evaluation of missing or potentially incorrect data (i.e. outliers) and generalizability [ 14 ]. This should be supplemented by a thorough evaluation of the potential data to inform the modeling work prior to its implementation, and ensuring that multiple modeling methods are applied.

This review found a wide variety of approaches, methods, statistical software and validation strategies that were employed in the application of machine learning methods to inform patient-provider decision making. Based on these findings, there is a need to ensure that multiple modeling approaches are employed in the development of machine learning-based models for patient care, which requires the highest research standards to reliably support shared evidence-based decision making. Models should be evaluated with clear criteria for model selection, and both internal and external validation are needed prior to applying these models to inform patient care. Few studies have yet to reach that bar of evidence to inform patient-provider decision making.

Availability of data and materials

All data generated or analyzed during this study are included in this published article and its supplementary information files.

Abbreviations

Artificial intelligence

Area under the curve

Classification and regression trees

Logistic least absolute shrinkage and selector operator

Steyerberg EW, Claggett B. Towards personalized therapy for multiple sclerosis: limitations of observational data. Brain. 2018;141(5):e38-e.

Fröhlich H, Balling R, Beerenwinkel N, Kohlbacher O, Kumar S, Lengauer T, et al. From hype to reality: data science enabling personalized medicine. BMC Med. 2018;16(1):150.

Article   PubMed   PubMed Central   Google Scholar  

Steyerberg EW. Clinical prediction models. Berlin: Springer; 2019.

Book   Google Scholar  

Schnabel RB, Sullivan LM, Levy D, Pencina MJ, Massaro JM, D’Agostino RB Sr, et al. Development of a risk score for atrial fibrillation (Framingham Heart Study): a community-based cohort study. Lancet. 2009;373(9665):739–45.

D’Agostino RB, Wolf PA, Belanger AJ, Kannel WB. Stroke risk profile: adjustment for antihypertensive medication. Framingham Study Stroke. 1994;25(1):40–3.

Article   CAS   PubMed   Google Scholar  

Framingham Heart Study: Risk Functions 2020. https://www.framinghamheartstudy.org/ .

Gawehn E, Hiss JA, Schneider G. Deep learning in drug discovery. Mol Inf. 2016;35:3–14.

Article   CAS   Google Scholar  

Vamathevan J, Clark D, Czodrowski P, Dunham I, Ferran E, Lee G, et al. Applications of machine learning in drug discovery and development. Nat Rev Drug Discov. 2019;18(6):463–77.

Article   CAS   PubMed   PubMed Central   Google Scholar  

Marcus G. Deep learning: A critical appraisal. arXiv preprint arXiv:180100631. 2018.

Grote T, Berens P. On the ethics of algorithmic decision-making in healthcare. J Med Ethics. 2020;46(3):205–11.

Article   PubMed   Google Scholar  

Brnabic A, Hess L, Carter GC, Robinson R, Araujo A, Swindle R. Methods used for the applicability of real-world data sources to individual patient decision making. Value Health. 2018;21:S102.

Article   Google Scholar  

Fu H, Zhou J, Faries DE. Estimating optimal treatment regimes via subgroup identification in randomized control trials and observational studies. Stat Med. 2016;35(19):3285–302.

Liang M, Ye T, Fu H. Estimating individualized optimal combination therapies through outcome weighted deep learning algorithms. Stat Med. 2018;37(27):3869–86.

Luo W, Phung D, Tran T, Gupta S, Rana S, Karmakar C, et al. Guidelines for developing and reporting machine learning predictive models in biomedical research: a multidisciplinary view. J Med Internet Res. 2016;18(12):e323.

Toussi M, Lamy J-B, Le Toumelin P, Venot A. Using data mining techniques to explore physicians’ therapeutic decisions when clinical guidelines do not provide recommendations: methods and example for type 2 diabetes. BMC Med Inform Decis Mak. 2009;9(1):28.

Ramezankhani A, Hadavandi E, Pournik O, Shahrabi J, Azizi F, Hadaegh F. Decision tree-based modelling for identification of potential interactions between type 2 diabetes risk factors: a decade follow-up in a Middle East prospective cohort study. BMJ Open. 2016;6(12):e013336.

Pei D, Zhang C, Quan Y, Guo Q. Identification of potential type II diabetes in a Chinese population with a sensitive decision tree approach. J Diabetes Res. 2019;2019:4248218.

Neefjes EC, van der Vorst MJ, Verdegaal BA, Beekman AT, Berkhof J, Verheul HM. Identification of patients with cancer with a high risk to develop delirium. Cancer Med. 2017;6(8):1861–70.

Mubeen AM, Asaei A, Bachman AH, Sidtis JJ, Ardekani BA, Initiative AsDN. A six-month longitudinal evaluation significantly improves accuracy of predicting incipient Alzheimer’s disease in mild cognitive impairment. J Neuroradiol. 2017;44(6):381–7.

Hische M, Luis-Dominguez O, Pfeiffer AF, Schwarz PE, Selbig J, Spranger J. Decision trees as a simple-to-use and reliable tool to identify individuals with impaired glucose metabolism or type 2 diabetes mellitus. Eur J Endocrinol. 2010;163(4):565.

Bertsimas D, Dunn J, Pawlowski C, Silberholz J, Weinstein A, Zhuo YD, et al. Applied informatics decision support tool for mortality predictions in patients with cancer. JCO Clin Cancer Inform. 2018;2:1–11.

Bannister CA, Halcox JP, Currie CJ, Preece A, Spasic I. A genetic programming approach to development of clinical prediction models: a case study in symptomatic cardiovascular disease. PLoS ONE. 2018;13(9):e0202685.

Article   PubMed   PubMed Central   CAS   Google Scholar  

Alaa AM, Bolton T, Di Angelantonio E, Rudd JHF, van der Schaar M. Cardiovascular disease risk prediction using automated machine learning: a prospective study of 423,604 UK Biobank participants. PLoS ONE. 2019;14(5):e0213653.

Baxter SL, Marks C, Kuo TT, Ohno-Machado L, Weinreb RN. Machine learning-based predictive modeling of surgical intervention in glaucoma using systemic data from electronic health records. Am J Ophthalmol. 2019;208:30–40.

Dong Y, Xu L, Fan Y, Xiang P, Gao X, Chen Y, et al. A novel surgical predictive model for Chinese Crohn’s disease patients. Medicine (Baltimore). 2019;98(46):e17510.

Hill NR, Ayoubkhani D, McEwan P, Sugrue DM, Farooqui U, Lister S, et al. Predicting atrial fibrillation in primary care using machine learning. PLoS ONE. 2019;14(11):e0224582.

Kang AR, Lee J, Jung W, Lee M, Park SY, Woo J, et al. Development of a prediction model for hypotension after induction of anesthesia using machine learning. PLoS ONE. 2020;15(4):e0231172.

Karhade AV, Ogink PT, Thio Q, Cha TD, Gormley WB, Hershman SH, et al. Development of machine learning algorithms for prediction of prolonged opioid prescription after surgery for lumbar disc herniation. Spine J. 2019;19(11):1764–71.

Kebede M, Zegeye DT, Zeleke BM. Predicting CD4 count changes among patients on antiretroviral treatment: Application of data mining techniques. Comput Methods Programs Biomed. 2017;152:149–57.

Kim I, Choi HJ, Ryu JM, Lee SK, Yu JH, Kim SW, et al. A predictive model for high/low risk group according to oncotype DX recurrence score using machine learning. Eur J Surg Oncol. 2019;45(2):134–40.

Kwon JM, Jeon KH, Kim HM, Kim MJ, Lim S, Kim KH, et al. Deep-learning-based out-of-hospital cardiac arrest prognostic system to predict clinical outcomes. Resuscitation. 2019;139:84–91.

Kwon JM, Lee Y, Lee Y, Lee S, Park J. An algorithm based on deep learning for predicting in-hospital cardiac arrest. J Am Heart Assoc. 2018;7(13):26.

Scheer JK, Smith JS, Schwab F, Lafage V, Shaffrey CI, Bess S, et al. Development of a preoperative predictive model for major complications following adult spinal deformity surgery. J Neurosurg Spine. 2017;26(6):736–43.

Lopez-de-Andres A, Hernandez-Barrera V, Lopez R, Martin-Junco P, Jimenez-Trujillo I, Alvaro-Meca A, et al. Predictors of in-hospital mortality following major lower extremity amputations in type 2 diabetic patients using artificial neural networks. BMC Med Res Methodol. 2016;16(1):160.

Rau H-H, Hsu C-Y, Lin Y-A, Atique S, Fuad A, Wei L-M, et al. Development of a web-based liver cancer prediction model for type II diabetes patients by using an artificial neural network. Comput Methods Programs Biomed. 2016;125:58–65.

Ng T, Chew L, Yap CW. A clinical decision support tool to predict survival in cancer patients beyond 120 days after palliative chemotherapy. J Palliat Med. 2012;15(8):863–9.

Pérez-Gandía C, Facchinetti A, Sparacino G, Cobelli C, Gómez E, Rigla M, et al. Artificial neural network algorithm for online glucose prediction from continuous glucose monitoring. Diabetes Technol Therapeut. 2010;12(1):81–8.

Azimi P, Mohammadi HR, Benzel EC, Shahzadi S, Azhari S. Use of artificial neural networks to decision making in patients with lumbar spinal canal stenosis. J Neurosurg Sci. 2017;61(6):603–11.

Bowman A, Rudolfer S, Weller P, Bland JDP. A prognostic model for the patient-reported outcome of surgical treatment of carpal tunnel syndrome. Muscle Nerve. 2018;58(6):784–9.

Hearn J, Ross HJ, Mueller B, Fan CP, Crowdy E, Duhamel J, et al. Neural networks for prognostication of patients with heart failure. Circ. 2018;11(8):e005193.

Google Scholar  

Isma’eel HA, Cremer PC, Khalaf S, Almedawar MM, Elhajj IH, Sakr GE, et al. Artificial neural network modeling enhances risk stratification and can reduce downstream testing for patients with suspected acute coronary syndromes, negative cardiac biomarkers, and normal ECGs. Int J Cardiovasc Imaging. 2016;32(4):687–96.

Isma’eel HA, Sakr GE, Serhan M, Lamaa N, Hakim A, Cremer PC, et al. Artificial neural network-based model enhances risk stratification and reduces non-invasive cardiac stress imaging compared to Diamond-Forrester and Morise risk assessment models: a prospective study. J Nucl Cardiol. 2018;25(5):1601–9.

Jovanovic P, Salkic NN, Zerem E. Artificial neural network predicts the need for therapeutic ERCP in patients with suspected choledocholithiasis. Gastrointest Endosc. 2014;80(2):260–8.

Zhou HF, Huang M, Ji JS, Zhu HD, Lu J, Guo JH, et al. Risk prediction for early biliary infection after percutaneous transhepatic biliary stent placement in malignant biliary obstruction. J Vasc Interv Radiol. 2019;30(8):1233-41.e1.

Hertroijs DF, Elissen AM, Brouwers MC, Schaper NC, Köhler S, Popa MC, et al. A risk score including body mass index, glycated haemoglobin and triglycerides predicts future glycaemic control in people with type 2 diabetes. Diabetes Obes Metab. 2018;20(3):681–8.

Oviedo S, Contreras I, Quiros C, Gimenez M, Conget I, Vehi J. Risk-based postprandial hypoglycemia forecasting using supervised learning. Int J Med Inf. 2019;126:1–8.

Khanji C, Lalonde L, Bareil C, Lussier MT, Perreault S, Schnitzer ME. Lasso regression for the prediction of intermediate outcomes related to cardiovascular disease prevention using the TRANSIT quality indicators. Med Care. 2019;57(1):63–72.

Anderson JP, Parikh JR, Shenfeld DK, Ivanov V, Marks C, Church BW, et al. Reverse engineering and evaluation of prediction models for progression to type 2 diabetes: an application of machine learning using electronic health records. J Diabetes Sci Technol. 2016;10(1):6–18.

Patsopoulos NA. A pragmatic view on pragmatic trials. Dialogues Clin Neurosci. 2011;13(2):217–24.

Lu CY. Observational studies: a review of study designs, challenges and strategies to reduce confounding. Int J Clin Pract. 2009;63(5):691–7.

Morgenstern H. Ecologic studies in epidemiology: concepts, principles, and methods. Annu Rev Public Health. 1995;16(1):61–81.

Vandenbroucke JP. In defense of case reports and case series. Ann Intern Med. 2001;134(4):330–4.

Buckland ST, Burnham KP, Augustin NH. Model selection: an integral part of inference. Biometrics. 1997;53:603–18.

Zagar A, Kadziola Z, Lipkovich I, Madigan D, Faries D. Evaluating bias control strategies in observational studies using frequentist model averaging 2020 (submitted).

Kang J, Schwartz R, Flickinger J, Beriwal S. Machine learning approaches for predicting radiation therapy outcomes: a clinician’s perspective. Int J Radiat Oncol Biol Phys. 2015;93(5):1127–35.

Scott IM, Lin W, Liakata M, Wood J, Vermeer CP, Allaway D, et al. Merits of random forests emerge in evaluation of chemometric classifiers by external validation. Anal Chim Acta. 2013;801:22–33.

Kuhn M, Johnson K. Applied predictive modeling. Berlin: Springer; 2013.

Hess L, Winfree K, Muehlenbein C, Zhu Y, Oton A, Princic N. Debunking Myths While Understanding Limitations. Am J Public Health. 2020;110(5):E2-E.

Thesmar D, Sraer D, Pinheiro L, Dadson N, Veliche R, Greenberg P. Combining the power of artificial intelligence with the richness of healthcare claims data: Opportunities and challenges. PharmacoEconomics. 2019;37(6):745–52.

Download references

Acknowledgements

Not applicable.

No funding was received for the conduct of this study.

Author information

Authors and affiliations.

Eli Lilly and Company, Sydney, NSW, Australia

Alan Brnabic

Eli Lilly and Company, Indianapolis, IN, USA

Lisa M. Hess

You can also search for this author in PubMed   Google Scholar

Contributions

AB and LMH contributed to the design, implementation, analysis and interpretation of the data included in this study. AB and LMH wrote, revised and finalized the manuscript for submission. AB and LMH have both read and approved the final manuscript.

Corresponding author

Correspondence to Lisa M. Hess .

Ethics declarations

Ethics approval and consent to participate, consent for publication, competing interests.

Authors are employees of Eli Lilly and Company and receive salary support in that role.

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Additional file 1..

Table S1. Study quality of eligible publications, modified Luo scale [14].

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Brnabic, A., Hess, L.M. Systematic literature review of machine learning methods used in the analysis of real-world data for patient-provider decision making. BMC Med Inform Decis Mak 21 , 54 (2021). https://doi.org/10.1186/s12911-021-01403-2

Download citation

Received : 07 July 2020

Accepted : 20 January 2021

Published : 15 February 2021

DOI : https://doi.org/10.1186/s12911-021-01403-2

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Machine learning
• Decision making
• Decision tree
• Random forest
• Automated neural network

BMC Medical Informatics and Decision Making

ISSN: 1472-6947

• General enquiries: [email protected]

Message placeholder

Literature Review of Machine Vision in Application Field

Chen Zhang 1 * , Xuewu Xu 1 ,2 , Chen Fan 1 and Guoping Wang 1

1 Mechanical Engineering, Xi’an Jiaotong University City College, Xi’an, Shaanxi Province, 710018, P. R. China 2 Xi’an Jiaotong University Intelligent Robot Innovation Institute, Xi’an, Shaanxi Province, 710018, P. R. China

* Corresponding author’s e-mail: [email protected]

Aiming at the application and research of machine vision, a comprehensive and detailed elaboration is carried out in its two application areas: visual inspection and robot vision. Introduce the composition, characteristics and application advantages of the machine vision system. Based on the analysis of the current research status at home and abroad, the application development trend of machine vision is prospected.

© The Authors, published by EDP Sciences 2021

Current usage metrics show cumulative count of Article Views (full-text article views including HTML views, PDF and ePub downloads, according to the available data) and Abstracts Views on Vision4Press platform.

Data correspond to usage on the plateform after 2015. The current usage metrics is available 48-96 hours after online publication and is updated daily on week days.

Initial download of the metrics may take a while.

The week's pick

Random Articles

• Extreme Learning Machine Models for Predicting Student Performance Feb 2021 Free Terminal Time in Optimal Control approach of Chikungunya Model April 2016 A Technique for Enhancement of Gray Image using Local Gamma Correction November 2014 An Enhanced Filtering Approach for High Density Salt and Pepper Noise to Restore Image with the Aid of Robust Estimator July 2012

Reseach Article

A literature review on machine vision based approaches for ripeness detection of fruits.

Rencheeraj Mohan, Sreekumar K. . A Literature Review on Machine Vision based Approaches for Ripeness Detection of Fruits. International Journal of Computer Applications. 182, 48 ( Apr 2019), 67-72. DOI=10.5120/ijca2019918744

Plump fruits are a vital part of the human diet giving mandatory vitamins, minerals and other health encouraging compounds. Quality assessment and finding of fruit ripeness is a major concern in agriculture business and becomes a growing research concern in computer vision. Image processing is an advanced field which led to a higher demand to reduce the high rate of errors and given more possible results. Therefore, the objective of many types of research is to standardize and reduce manual work in the classification of tomatoes ripeness. One of the most important feature of an image is color. Estimating the ripeness of fruits via color can be performed as it is the dominant feature in describing the information of the image. However, each color models have been given a different performance when used in the experiment. This paper is a survey of different techniques that are deployed over different varieties of fruit images in order to detect maturity stages for ripening, fruit region estimation and also, the effect of different color models and other features on detecting ripeness was studied in this literature survey.

• Wu, Jingui, et al. "Automatic Recognition of Ripening Tomatoes by Combining Multi-Feature Fusion with a Bi-Layer Classification Strategy for Harvesting Robots." Sensors 19.3 (2019): 612.
• Tan, Kezhu, et al. "Recognising blueberry fruit of different maturity using histogram oriented gradients and colour features in outdoor scenes." Biosystems Engineering 176 (2018): 59-72.
• Zhang, Yan, et al. "Deep indicator for fine-grained classification of banana’s ripening stages." EURASIP Journal on Image and Video Processing 2018.1 (2018): 46.
• UluiŞik, Selman, FikretYildiz, and AhmetTuranÖzdemİr. "Image processing based machine vision system for tomato volume estimation." 2018 Electric Electronics, Computer Science, Biomedical Engineerings' Meeting (EBBT). IEEE, 2018.
• Kumar, Arun, Vijay S. Rajpurohit, and Bhairu J. Jirage. "Pomegranate fruit quality assessment using machine intelligence and wavelet features." Journal of Horticultural Research 26.1 (2018): 53-60.
• Taofik, A., et al. "Design of Smart System to Detect Ripeness of Tomato and Chili with New Approach in Data Acquisition." IOP Conference Series: Materials Science and Engineering. Vol. 288. No. 1. IOP Publishing, 2018..
• Mim, Farjana Sultana, et al. "Automatic detection of mango ripening stages–An application of information technology to botany." ScientiaHorticulturae 237 (2018): 156-163.
• Pereira, Luiz Fernando Santos, et al. "Predicting the ripening of papaya fruit with digital imaging and random forests." Computers and Electronics in Agriculture 145 (2018): 76-82.
• Tu, Shuqin, et al. "Detection of passion fruits and maturity classification using Red-Green-Blue Depth images." Biosystems Engineering 175 (2018): 156-167.
• Wan, Peng, et al. "A methodology for fresh tomato maturity detection using computer vision." Computers and electronics in agriculture 146 (2018): 43-50.
• Li, Bairong, Yan Long, and Huaibo Song. "Detection of green apples in natural scenes based on saliency theory and Gaussian curve fitting." International Journal of Agricultural and Biological Engineering 11.1 (2018): 192-198.

Index Terms

Computer Science Information Sciences

Fruit classification maturity detection segmentation classification feature extraction. Ripeness detection.

Share this:

• Click to share on Twitter (Opens in new window)
• Click to share on LinkedIn (Opens in new window)
• Click to share on Facebook (Opens in new window)

Artificial Intelligence for Literature Reviews: Opportunities and Challenges

Exciting news! Our new survey paper about AI tools for literature reviews was published by Artificial Intelligence Review .

The paper presents a comprehensive review of the use of Artificial Intelligence (AI) in Systematic Literature Reviews (SLRs). The increasing role of AI in this field shows great potential in providing more effective support for researchers, moving towards the semi-automatic creation of literature reviews. Our study focuses on how AI techniques are applied in the semi-automation of SLRs, specifically in the screening and extraction phases. We examine 21 leading SLR tools using a framework that combines 23 traditional features with 11 AI features. We also analyse 11 recent tools that leverage large language models for searching the literature and assisting academic writing. Finally, the paper discusses current trends in the field, outlines key research challenges, and suggests directions for future research. We highlight three primary research challenges: integrating advanced AI solutions, such as large language models and knowledge graphs, improving usability, and developing a standardised evaluation framework. We also propose best practices to ensure more robust evaluations in terms of performance, usability, and transparency.

Francisco Bolanos, Angelo Salatino, Francesco Osborne, Enrico Motta

Follow us on Twitter

@ERLrobotleague

Information

• Author Services

Initiatives

You are accessing a machine-readable page. In order to be human-readable, please install an RSS reader.

All articles published by MDPI are made immediately available worldwide under an open access license. No special permission is required to reuse all or part of the article published by MDPI, including figures and tables. For articles published under an open access Creative Common CC BY license, any part of the article may be reused without permission provided that the original article is clearly cited. For more information, please refer to https://www.mdpi.com/openaccess .

Feature papers represent the most advanced research with significant potential for high impact in the field. A Feature Paper should be a substantial original Article that involves several techniques or approaches, provides an outlook for future research directions and describes possible research applications.

Feature papers are submitted upon individual invitation or recommendation by the scientific editors and must receive positive feedback from the reviewers.

Editor’s Choice articles are based on recommendations by the scientific editors of MDPI journals from around the world. Editors select a small number of articles recently published in the journal that they believe will be particularly interesting to readers, or important in the respective research area. The aim is to provide a snapshot of some of the most exciting work published in the various research areas of the journal.

Original Submission Date Received: .

• Active Journals
• Find a Journal
• Proceedings Series
• For Authors
• For Reviewers
• For Editors
• For Librarians
• For Publishers
• For Societies
• For Conference Organizers
• Open Access Policy
• Institutional Open Access Program
• Special Issues Guidelines
• Editorial Process
• Research and Publication Ethics
• Article Processing Charges
• Testimonials
• Preprints.org
• SciProfiles
• Encyclopedia

Article Menu

• Subscribe SciFeed
• Google Scholar
• on Google Scholar
• Table of Contents

Find support for a specific problem in the support section of our website.

Please let us know what you think of our products and services.

Visit our dedicated information section to learn more about MDPI.

JSmol Viewer

Atezolizumab-induced ulcerative colitis in patient with hepatocellular carcinoma: case report and literature review.

1. Introduction

2. case presentation, 3. discussion, 4. conclusions, author contributions, institutional review board statement, informed consent statement, data availability statement, conflicts of interest.

• Liu, C.; Yang, M.; Zhang, D.; Chen, M.; Zhu, D. Clinical cancer immunotherapy: Current progress and prospects. Front. Immunol. 2022 , 13 , 961805. [ Google Scholar ] [ CrossRef ]
• Lee, Y.R. A multidisciplinary approach with immunotherapies for advanced hepatocellular carcinoma. J. Liver Cancer 2023 , 23 , 316–329. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Finn, R.S.; Qin, S.; Ikeda, M.; Galle, P.R.; Ducreux, M.; Kim, T.-Y.; Kudo, M.; Breder, V.; Merle, P.; Kaseb, A.O.; et al. Atezolizumab plus Bevacizumab in Unresectable Hepatocellular Carcinoma. N. Engl. J. Med. 2020 , 382 , 1894–1905. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Korean Liver Cancer Association; National Cancer Center (NCC) Korea. 2022 KLCA-NCC Korea practice guidelines for the management of hepatocellular carcinoma. J Liver Cancer 2023 , 23 , 1–120. [ Google Scholar ] [ CrossRef ]
• Martins, F.; Sofiya, L.; Sykiotis, G.P.; Lamine, F.; Maillard, M.; Fraga, M.; Shabafrouz, K.; Ribi, C.; Cairoli, A.; Guex-Crosier, Y.; et al. Adverse effects of immune-checkpoint inhibitors: Epidemiology, management and surveillance. Nat. Rev. Clin. Oncol. 2019 , 16 , 563–580. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Song, Y.G.; Yoo, J.J.; Kim, S.G.; Kim, Y.S. Complications of immunotherapy in advanced hepatocellular carcinoma. J. Liver Cancer 2024 , 24 , 9–16. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Song, Y.-G.; Yeom, K.-M.; Jung, E.A.; Kim, S.G.; Kim, Y.S.; Yoo, J.-J. Risk of Bleeding in Hepatocellular Carcinoma Patients Treated with Atezolizumab/Bevacizumab: A Systematic Review and Meta-Analysis. Liver Cancer 2024 , 1–11. [ Google Scholar ] [ CrossRef ]
• Pallin, D.J.; Baugh, C.W.; Postow, M.A.; Caterino, J.M.; Erickson, T.B.; Lyman, G.H. Immune-related Adverse Events in Cancer Patients. Acad. Emerg. Med. 2018 , 25 , 819–827. [ Google Scholar ] [ CrossRef ]
• Som, A.; Mandaliya, R.; Alsaadi, D.; Farshidpour, M.; Charabaty, A.; Malhotra, N.; Mattar, M.C. Immune checkpoint inhibitor-induced colitis: A comprehensive review. World J. Clin. Cases 2019 , 7 , 405–418. [ Google Scholar ] [ CrossRef ]
• Yu, K.; Mathew, A.; Abraham, F.; Amin, R.; Kono, M.; Overman, M.; Zhao, D.; Khan, A.; Khan, M.A.; Thomas, A.S.; et al. Immune checkpoint inhibitor-related gastrointestinal toxicity in patients with malignancy involving the luminal gastrointestinal tract and its impact on cancer outcomes. Ann. Gastroenterol. 2022 , 35 , 514–521. [ Google Scholar ] [ CrossRef ]
• Yamauchi, R.; Araki, T.; Mitsuyama, K.; Tokito, T.; Ishii, H.; Yoshioka, S.; Kuwaki, K.; Mori, A.; Yoshimura, T.; Tsuruta, O.; et al. The characteristics of nivolumab-induced colitis: An evaluation of three cases and a literature review. BMC Gastroenterol. 2018 , 18 , 135. [ Google Scholar ] [ CrossRef ]
• Steiger, S.; Marcucci, V.; Desai, V.; Zheng, M.; Parker, G. Refractory Fulminant Colitis Requiring Surgical Intervention in a Patient With Ulcerative Colitis on Atezolizumab Therapy for Small Cell Lung Cancer: An Atypical Case. Cureus 2022 , 14 , e25437. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Pentapati, S.; Caucci, S.; Balmuri, S.; Devarkonda, V. Complete Remission of Advanced Hepatocellular Carcinoma in a Patient with Ulcerative Colitis Treated With Atezolizumab and Bevacizumab: A Case Report. Cureus 2023 , 15 , e39538. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Kubo, K.; Kato, M.; Mabe, K. Nivolumab-Associated Colitis Mimicking Ulcerative Colitis. Clin. Gastroenterol. Hepatol. 2017 , 15 , A35–A36. [ Google Scholar ] [ CrossRef ]
• Fuji, T.; Arai, J.; Otoyama, Y.; Nio, Y.; Sugiura, I.; Nakajima, Y.; Kajiwara, A.; Ichikawa, Y.; Uozumi, S.; Shimozuma, Y.; et al. A Case of Hepatocellular Carcinoma Successfully Resumed Atezolizumab and Bevacizumab After Associated Grade 3 Diarrhea and Grade 2 Colitis: Case Report and Literature Review. Onco Targets Ther. 2022 , 15 , 1281–1288. [ Google Scholar ] [ CrossRef ]
• Gallo, A.; Talerico, R.; Novello, L.; Giustiniani, M.C.; D’Argento, E.; Bria, E.; Montalto, M. Collagenous colitis and atezolizumab therapy: An atypical case. Clin. J. Gastroenterol. 2021 , 14 , 165–169. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Tang, L.; Wang, J.; Lin, N.; Zhou, Y.; He, W.; Liu, J.; Ma, X. Immune Checkpoint Inhibitor-Associated Colitis: From Mechanism to Management. Front. Immunol. 2021 , 12 , 800879. [ Google Scholar ] [ CrossRef ]
• Hone Lopez, S.; Kats-Ugurlu, G.; Renken, R.J.; Buikema, H.J.; de Groot, M.R.; Visschedijk, M.C.; Dijkstra, G.; Jalving, M.; de Haan, J.J. Immune checkpoint inhibitor treatment induces colitis with heavy infiltration of CD8 + T cells and an infiltration pattern that resembles ulcerative colitis. Virchows Arch. 2021 , 479 , 1119–1129. [ Google Scholar ] [ CrossRef ]
• Nielsen, D.L.; Juhl, C.B.; Chen, I.M.; Kellermann, L.; Nielsen, O.H. Immune checkpoint Inhibitor-Induced diarrhea and Colitis: Incidence and Management. A systematic review and Meta-analysis. Cancer Treat. Rev. 2022 , 109 , 102440. [ Google Scholar ] [ CrossRef ]
• Zou, F.; Wang, X.; Glitza Oliva, I.C.; McQuade, J.L.; Wang, J.; Zhang, H.C.; Thompson, J.A.; Thomas, A.S.; Wang, Y. Fecal calprotectin concentration to assess endoscopic and histologic remission in patients with cancer with immune-mediated diarrhea and colitis. J. Immunother. Cancer 2021 , 9 , e002058. [ Google Scholar ] [ CrossRef ]
• Nishida, T.; Iijima, H.; Adachi, S. Immune checkpoint inhibitor-induced diarrhea/colitis: Endoscopic and pathologic findings. World J. Gastrointest. Pathophysiol. 2019 , 10 , 17–28. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Grover, S.; Wang, Y.; Dougan, M. Immune Checkpoint Inhibitor Colitis ; UpToDate: Waltham, MA, USA, 2024. [ Google Scholar ]
• Bellaguarda, E.; Hanauer, S. Checkpoint Inhibitor-Induced Colitis. Am. J. Gastroenterol. 2020 , 115 , 202–210. [ Google Scholar ] [ CrossRef ]
• Sasson, S.C.; Slevin, S.M.; Cheung, V.T.F.; Nassiri, I.; Olsson-Brown, A.; Fryer, E.; Ferreira, R.C.; Trzupek, D.; Gupta, T.; Al-Hillawi, L.; et al. Interferon-Gamma-Producing CD8(+) Tissue Resident Memory T Cells Are a Targetable Hallmark of Immune Checkpoint Inhibitor-Colitis. Gastroenterology 2021 , 161 , 1229–1244.e1229. [ Google Scholar ] [ CrossRef ]
• Dougan, M.; Wang, Y.; Rubio-Tapia, A.; Lim, J.K. AGA Clinical Practice Update on Diagnosis and Management of Immune Checkpoint Inhibitor Colitis and Hepatitis: Expert Review. Gastroenterology 2021 , 160 , 1384–1393. [ Google Scholar ] [ CrossRef ]
• Geukes Foppen, M.H.; Rozeman, E.A.; van Wilpe, S.; Postma, C.; Snaebjornsson, P.; van Thienen, J.V.; van Leerdam, M.E.; van den Heuvel, M.; Blank, C.U.; van Dieren, J.; et al. Immune checkpoint inhibition-related colitis: Symptoms, endoscopic features, histology and response to management. ESMO Open 2018 , 3 , e000278. [ Google Scholar ] [ CrossRef ]
• Johnson, D.H.; Zobniw, C.M.; Trinh, V.A.; Ma, J.; Bassett, R.L., Jr.; Abdel-Wahab, N.; Anderson, J.; Davis, J.E.; Joseph, J.; Uemura, M.; et al. Infliximab associated with faster symptom resolution compared with corticosteroids alone for the management of immune-related enterocolitis. J. Immunother. Cancer 2018 , 6 , 103. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Abu-Sbeih, H.; Ali, F.S.; Alsaadi, D.; Jennings, J.; Luo, W.; Gong, Z.; Richards, D.M.; Charabaty, A.; Wang, Y. Outcomes of vedolizumab therapy in patients with immune checkpoint inhibitor-induced colitis: A multi-center study. J. Immunother. Cancer 2018 , 6 , 142. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Thomas Anusha, S.; Ma, W.; Wang, Y. Ustekinumab for Refractory Colitis Associated with Immune Checkpoint Inhibitors. N. Engl. J. Med. 2021 , 384 , 581–583. [ Google Scholar ] [ CrossRef ]
• Céspedes Martínez, E.; Robles Alonso, V.; Herrera-De Guise, C.; Mayorga, L.; Casellas, F.; Roca-Herrera, M.; Borruel, N. Severe and refractory gastrointestinal toxicity due to immune checkpoint inhibitors: Clinical experience in a tertiary referral hospital. Rev. Esp. Enferm. Dig. 2023 , 115 , 567–573. [ Google Scholar ] [ CrossRef ]
• Fernández-Gordón Sánchez, F.M.; Gómez-Domínguez, E.; Paredes Ruiz, D.; Rodríguez Gil, Y.; Martín Algíbez, A.; Fernández Vázquez, I.; Martínez Montiel, P. Ustekinumab for corticodependent immune-mediated colitis by pembrolizumab, an alternative for patients with concomitant liver injury. Rev. Esp. Enferm. Dig. 2022 , 114 , 356–357. [ Google Scholar ] [ CrossRef ]
• Halsey, T.M.; Thomas, A.S.; Hayase, T.; Ma, W.; Abu-Sbeih, H.; Sun, B.; Parra, E.R.; Jiang, Z.D.; DuPont, H.L.; Sanchez, C.; et al. Microbiome alteration via fecal microbiota transplantation is effective for refractory immune checkpoint inhibitor-induced colitis. Sci. Transl. Med. 2023 , 15 , eabq4006. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Burla, J.; Bluemel, S.; Biedermann, L.; Barysch, M.J.; Dummer, R.; Levesque, M.P.; Gubler, C.; Morell, B.; Rogler, G.; Scharl, M. Retrospective Analysis of Treatment and Complications of Immune Checkpoint Inhibitor-Associated Colitis: Histological Ulcerations as Potential Predictor for a Steroid-Refractory Disease Course. Inflamm. Intest. Dis. 2020 , 5 , 109–116. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Abu-Sbeih, H.; Ali, F.S.; Naqash, A.R.; Owen, D.H.; Patel, S.; Otterson, G.A.; Kendra, K.; Ricciuti, B.; Chiari, R.; De Giglio, A.; et al. Resumption of Immune Checkpoint Inhibitor Therapy After Immune-Mediated Colitis. J. Clin. Oncol. 2019 , 37 , 2738–2745. [ Google Scholar ] [ CrossRef ] [ PubMed ]

Click here to enlarge figure

Share and Cite

Kim, H.; Shin, Y.E.; Yoo, H.-J.; Kim, J.-Y.; Yoo, J.-J.; Kim, S.G.; Kim, Y.S. Atezolizumab-Induced Ulcerative Colitis in Patient with Hepatocellular Carcinoma: Case Report and Literature Review. Medicina 2024 , 60 , 1422. https://doi.org/10.3390/medicina60091422

Kim H, Shin YE, Yoo H-J, Kim J-Y, Yoo J-J, Kim SG, Kim YS. Atezolizumab-Induced Ulcerative Colitis in Patient with Hepatocellular Carcinoma: Case Report and Literature Review. Medicina . 2024; 60(9):1422. https://doi.org/10.3390/medicina60091422

Kim, Hyuk, Yoon E Shin, Hye-Jin Yoo, Jae-Young Kim, Jeong-Ju Yoo, Sang Gyune Kim, and Young Seok Kim. 2024. "Atezolizumab-Induced Ulcerative Colitis in Patient with Hepatocellular Carcinoma: Case Report and Literature Review" Medicina 60, no. 9: 1422. https://doi.org/10.3390/medicina60091422

Article Metrics

Further information, mdpi initiatives, follow mdpi.

Subscribe to receive issue release notifications and newsletters from MDPI journals