-
Notifications
You must be signed in to change notification settings - Fork 4
Expand file tree
/
Copy pathmarbleflow.html
More file actions
151 lines (134 loc) · 14.5 KB
/
Copy pathmarbleflow.html
File metadata and controls
151 lines (134 loc) · 14.5 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
<!DOCTYPE html>
<html lang="en-US">
<head>
<title>Marble Flow</title>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<link rel="alternate" type="application/rss+xml" title="RSS Feed" href="rss.xml">
<!-- SEO Description -->
<meta name="author" content="Nicholas S. Selby">
<meta name="description" content="Optical Flow and Object Tracking for Robots">
<link rel="canonical" href="https://rupumped.github.io/marbleflow.html">
<!-- Open Graph Tags -->
<meta property="og:title" content="Marble Flow">
<meta property="og:description" content="Optical Flow and Object Tracking for Robots">
<meta property="og:image" content="https://rupumped.github.io/project-pages/marbleflow-cover.jpg">
<meta property="og:image:width" content="1792">
<meta property="og:image:height" content="717">
<meta property="og:image:alt" content="A robot playing with marbles">
<meta property="og:url" content="https://rupumped.github.io/marbleflow.html">
<meta property="og:type" content="website">
<meta property="og:video" content="https://www.youtube.com/watch?v=sgq_YhP6NFc">
<!-- Favicon -->
<link rel="icon" href="./favicon.svg" type="image/svg+xml">
<link rel="icon" href="./favicon.ico" sizes="any">
<!-- Fonts -->
<link rel="preconnect" href="https://fonts.googleapis.com">
<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin>
<link href="https://fonts.googleapis.com/css2?family=Bricolage+Grotesque:opsz,wght@12..96,200..800&family=Literata:ital,opsz,wght@0,7..72,300..700;1,7..72,300..700&display=swap" rel="stylesheet">
<!-- Custom CSS -->
<link rel="stylesheet" type="text/css" href="main.css">
<link rel="stylesheet" type="text/css" href="secondary.css">
<link rel="stylesheet" type="text/css" href="project-pages/project-page.css">
</head>
<body>
<header>
<div class="header-content">
<a href="index.html" id="name">NICHOLAS S SELBY</a>
<nav>
<input class="menu-btn" type="checkbox" id="menu-btn">
<label class="menu-icon" for="menu-btn" tabindex="0"><span class="navicon"></span></label>
<ul class="menu">
<li><a href="index.html">HOME</a></li>
<li><a href="about.html">ABOUT</a></li>
<li><a href="selected-work.html">PROJECTS</a></li>
<li><a href="scholarship.html">SCHOLARSHIP</a></li>
<li><a href="service.html">SERVICE</a></li>
<li><a href="blog.html">BLOG</a></li>
</ul>
</nav>
</div>
</header>
<main>
<h1>Optical Flow and Object Tracking for Robots</h1>
<img loading="lazy" class="img-full-width" src="project-pages/marbleflow-cover.jpg" alt="A robot playing with marbles">
<h2 class="challenge"><b>Challenge:</b> Program a robot arm with a camera to follow objects</h2>
<div class="separator"></div>
<div class="centered section">
<iframe width="560" height="315" style="margin: auto;" src="https://www.youtube-nocookie.com/embed/sgq_YhP6NFc" title="YouTube video player" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
</div>
<p>For our graduate-level computer vision course final project, I worked with <a rel="external" target="_blank" href="https://www.linkedin.com/in/abbasmshikari/">Abbas Shikari</a> to develop MarbleFlow, an innovative project aimed at developing an optical marble tracker using a mobile camera. The project involved creating a system where a camera mounted on a robot arm could track a ball's movement, enabling the arm to follow the ball based on camera input. Additionally, we implemented dense optical flow for estimating the camera's motion trajectory. Our project implemented two key computer vision algorithms:</p>
<ol>
<li><b>Ball Tracking System:</b> Leveraging threshold-based object detection, we calculated the 2D centroid position of a ball from a video feed. The system provided an error signal in x and y axes by comparing the centroid position with the image center, which was used for a velocity controller on the Sawyer robot arm.</li>
<li><b>Dense Optical Flow Algorithm:</b> Implemented on the full image, this algorithm averaged the dense optical flow to estimate the camera's velocity, allowing the recreation of the camera's estimated trajectory.</li>
</ol>
<img loading="lazy" class="img-center" style="background-color: lightgray;" src="https://raw.githubusercontent.com/ashikari/MarbleFlow/master/block_diagram.png" alt="A block diagram illustrating our software. Video feed from the camera is fed through a series of transforms. Next, we use blob detection to compute the marble's location relative to the center of the image. Then, we feed the location into a PID controller and use an inverse kinematics model to command the robot to move to minimize the distance between the center of the image and the marble's location.">
<p>We wrote the computer vision algorithms in C++ using OpenCV and employed the ROS framework in Python to implement the robot PID controller and kinematics. The project successfully demonstrated the capability to track a ball and control a robotic arm based on visual inputs. The dense optical flow algorithm provided accurate camera motion estimates, enhancing the system's overall efficacy.</p>
<p>MarbleFlow was a rewarding educational experience that allowed us to deepen our understanding of fundamental computer vision algorithms and their practical applications. It provided us with hands-on experience in integrating hardware and software to solve a specific problem: tracking and following a moving object with a robotic arm. We are grateful for the opportunity to have worked on this project and are excited to apply the skills and knowledge we've gained in our future endeavors in the field of computer vision and robotics. Download our code and try it yourself on <a rel="external" target="_blank" href="https://github.com/ashikari/MarbleFlow">our GitHub repository</a>.</p>
<section id="skills-demonstrated">
<h2>Skills Demonstrated</h2>
<div class="skills">
<div class="skill">Robotics & Control</div>
<div class="skill">Computer Vision</div>
<div class="skill">Python</div>
<div class="skill">C++</div>
</div>
</section>
</main>
<footer>
<div id="top-footer">
<div id="footer-socials">
<a aria-label="My LinkedIn profile" rel="external" href="https://www.linkedin.com/in/nicholas-selby-5278b334/" target="_blank">
<svg viewBox="0 0 448 512" xmlns="http://www.w3.org/2000/svg" aria-labelledby="LinkedIn-title LinkedIn-desc">
<title id="LinkedIn-title">LinkedIn</title>
<desc id="LinkedIn-desc">LinkedIn icon</desc>
<path d="m416 32h-384.1c-17.6 0-31.9 14.5-31.9 32.3v383.4c0 17.8 14.3 32.3 31.9 32.3h384.1c17.6 0 32-14.5 32-32.3v-383.4c0-17.8-14.4-32.3-32-32.3zm-280.6 384h-66.4v-213.8h66.5v213.8zm-33.2-243c-21.3 0-38.5-17.3-38.5-38.5s17.2-38.5 38.5-38.5c21.2 0 38.5 17.3 38.5 38.5 0 21.3-17.2 38.5-38.5 38.5zm282.1 243h-66.4v-104c0-24.8-.5-56.7-34.5-56.7-34.6 0-39.9 27-39.9 54.9v105.8h-66.4v-213.8h63.7v29.2h.9c8.9-16.8 30.6-34.5 62.9-34.5 67.2 0 79.7 44.3 79.7 101.9z"/>
</svg>
</a>
<a aria-label="My Google Scholar Profile" rel="external" href="https://scholar.google.com/citations?user=SKcs1pEAAAAJ" target="_blank">
<svg viewBox="0 0 448 512" xmlns="http://www.w3.org/2000/svg" aria-labelledby="Google-Scholar-title Google-Scholar-desc">
<title id="Google-Scholar-title">Google Scholar</title>
<desc id="Google-Scholar-desc">Google Scholar icon</desc>
<path d="m48 32c-26.5 0-48 21.5-48 48v352c0 26.5 21.5 48 48 48h352c26.5 0 48-21.5 48-48v-352c0-26.5-21.5-48-48-48zm140.69531 64h178.24024l-16.47071 12.85938v22.69335c6.1159.78532 5.46875 4.46019 5.46875 8.85743v107.22265c0 4.96563-4.06205 9.0293-9.02734 9.0293h-3.32422c-4.96563 0-9.0293-4.06403-9.0293-9.0293v-107.22265c0-4.40781-.63995-8.08659 5.52149-8.86133v-14.26563l-47.42383 38.89258c.54795 1.01573 1.06775 1.6675 1.5625 2.51953 4.16684 7.37735 6.28906 16.54919 6.28906 27.75977 0 8.59325-1.43204 16.31497-4.33593 23.13086-2.89082 6.81736-6.4066 12.38451-10.50782 16.67383-4.10119 4.30429-8.21545 8.23614-12.33008 11.77734-4.11326 3.54813-7.62906 7.24664-10.50781 11.08789-2.90139 3.82814-4.34765 7.78747-4.34765 11.88867 0 4.10814 1.87594 8.28085 5.61328 12.48633 3.72386 4.2186 8.30529 8.30635 13.72265 12.34375 5.42938 4.01043 10.84634 8.46278 16.26368 13.30664 5.42904 4.83584 9.98706 11.06107 13.71093 18.62109 3.75047 7.58518 5.625 15.93732 5.625 25.11719 0 12.10928-3.08756 23.04599-9.24609 32.7793-6.17204 9.69317-14.21877 17.42887-24.10156 23.09961-9.90904 5.70939-20.50722 10.00631-31.8086 12.91015-11.32797 2.87733-22.56501 4.32227-33.78906 4.32227-7.08423 0-14.23265-.5475-21.42187-1.66602-7.21219-1.12-14.43823-3.09828-21.7168-5.88476-7.29167-2.80655-13.75106-6.25079-19.34961-10.39063-5.61205-4.09467-10.12984-9.38173-13.59375-15.82031-3.46392-6.43852-5.18359-13.67857-5.18359-21.71875 0-9.53863 2.65694-18.38532 7.98242-26.63476 5.32512-8.18928 12.3823-15.02552 21.1582-20.44141 15.31325-9.52549 39.33645-15.4113 72.03125-17.63672-7.47326-9.34283-11.22266-18.13975-11.22266-26.36914 0-4.68271 1.22186-9.69984 3.64454-15.11719-3.90652.54687-7.9286.85352-12.03125.85352-17.56273 0-32.39541-5.70952-44.43946-17.20703-12.04402-11.47856-18.05859-25.8462-18.05859-43.2168 0-1.81599.05219-3.41866.18164-5.18945h-71.378907zm31.37891 38.33398c-11.19892 0-19.79186 4.02906-25.78125 12.07032-5.9879 8.02012-8.98438 17.73465-8.98438 29.14648 0 9.71981 1.64061 19.61644 4.92188 29.70703 3.2681 10.08548 8.62123 19.08948 16.12109 27.02539 7.47324 7.95594 16.17192 11.92578 26.05469 11.92578 11.01591 0 19.62286-3.68944 25.79492-11.07226 6.14505-7.37043 9.23047-16.67343 9.23047-27.87891 0-9.54517-1.6264-19.537-4.89453-29.98633-3.25608-10.46247-8.68505-19.90262-16.26367-28.30664-7.55234-8.4321-16.30333-12.63086-26.19922-12.63086zm18.51758 172.08008c-8.03469 0-15.93905.71228-23.69922 2.13086-7.76017 1.37996-15.41675 3.70986-22.96875 6.99805-7.57861 3.26958-13.69819 8.07499-18.35938 14.42969-4.68784 6.34667-7.01953 13.82169-7.01953 22.42187 0 8.19582 2.05913 15.50628 6.17383 21.87305 4.10265 6.32768 9.51795 11.28394 16.25 14.83203 6.73201 3.55471 13.78885 6.23613 21.1582 8 7.3828 1.75184 14.89682 2.66406 22.55274 2.66406 15.14262 0 28.17578-3.41115 39.11328-10.22851 10.91054-6.8159 16.38086-17.34206 16.38086-31.54883 0-2.98812-.41689-5.93201-1.23828-8.80274-.85861-2.90389-1.70365-5.3892-2.53711-7.45312-.83345-2.02525-2.42173-4.4673-4.76563-7.2793-2.33081-2.81861-4.11569-4.90255-5.31445-6.28906-1.2239-1.43208-3.51215-3.47703-6.88672-6.17969-3.34426-2.71375-5.48032-4.40548-6.43164-5.03125h-.00195c-.93771-.65762-3.3717-2.4355-7.29102-5.32031-3.9189-2.91084-6.06651-4.45407-6.44531-4.63672-2.05736-.37077-4.94607-.58008-8.66992-.58008z"/>
</svg>
</a>
<a aria-label="My GitHub profile" rel="external" href="https://github.com/rupumped" target="_blank">
<svg viewBox="0 0 448 512" xmlns="http://www.w3.org/2000/svg" aria-labelledby="GitHub-title GitHub-desc">
<title id="GitHub-title">GitHub</title>
<desc id="GitHub-desc">GitHub icon</desc>
<path d="M400 32H48C21.5 32 0 53.5 0 80v352c0 26.5 21.5 48 48 48h352c26.5 0 48-21.5 48-48V80c0-26.5-21.5-48-48-48zM277.3 415.7c-8.4 1.5-11.5-3.7-11.5-8 0-5.4.2-33 .2-55.3 0-15.6-5.2-25.5-11.3-30.7 37-4.1 76-9.2 76-73.1 0-18.2-6.5-27.3-17.1-39 1.7-4.3 7.4-22-1.7-45-13.9-4.3-45.7 17.9-45.7 17.9-13.2-3.7-27.5-5.6-41.6-5.6-14.1 0-28.4 1.9-41.6 5.6 0 0-31.8-22.2-45.7-17.9-9.1 22.9-3.5 40.6-1.7 45-10.6 11.7-15.6 20.8-15.6 39 0 63.6 37.3 69 74.3 73.1-4.8 4.3-9.1 11.7-10.6 22.3-9.5 4.3-33.8 11.7-48.3-13.9-9.1-15.8-25.5-17.1-25.5-17.1-16.2-.2-1.1 10.2-1.1 10.2 10.8 5 18.4 24.2 18.4 24.2 9.7 29.7 56.1 19.7 56.1 19.7 0 13.9.2 36.5.2 40.6 0 4.3-3 9.5-11.5 8-66-22.1-112.2-84.9-112.2-158.3 0-91.8 70.2-161.5 162-161.5S388 165.6 388 257.4c.1 73.4-44.7 136.3-110.7 158.3zm-98.1-61.1c-1.9.4-3.7-.4-3.9-1.7-.2-1.5 1.1-2.8 3-3.2 1.9-.2 3.7.6 3.9 1.9.3 1.3-1 2.6-3 3zm-9.5-.9c0 1.3-1.5 2.4-3.5 2.4-2.2.2-3.7-.9-3.7-2.4 0-1.3 1.5-2.4 3.5-2.4 1.9-.2 3.7.9 3.7 2.4zm-13.7-1.1c-.4 1.3-2.4 1.9-4.1 1.3-1.9-.4-3.2-1.9-2.8-3.2.4-1.3 2.4-1.9 4.1-1.5 2 .6 3.3 2.1 2.8 3.4zm-12.3-5.4c-.9 1.1-2.8.9-4.3-.6-1.5-1.3-1.9-3.2-.9-4.1.9-1.1 2.8-.9 4.3.6 1.3 1.3 1.8 3.3.9 4.1zm-9.1-9.1c-.9.6-2.6 0-3.7-1.5s-1.1-3.2 0-3.9c1.1-.9 2.8-.2 3.7 1.3 1.1 1.5 1.1 3.3 0 4.1zm-6.5-9.7c-.9.9-2.4.4-3.5-.6-1.1-1.3-1.3-2.8-.4-3.5.9-.9 2.4-.4 3.5.6 1.1 1.3 1.3 2.8.4 3.5zm-6.7-7.4c-.4.9-1.7 1.1-2.8.4-1.3-.6-1.9-1.7-1.5-2.6.4-.6 1.5-.9 2.8-.4 1.3.7 1.9 1.8 1.5 2.6z"/>
</svg>
</a>
<a aria-label="My RSS feed" href="rss.html">
<svg height="800px" width="800px" version="1.1" id="Layer_1" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" viewBox="-143 145 512 512" xml:space="preserve" aria-labelledby="RSS-title RSS-desc">
<title id="RSS-title">RSS</title>
<desc id="RSS-desc">RSS icon</desc>
<path d="M329,145h-432c-22.1,0-40,17.9-40,40v432c0,22.1,17.9,40,40,40h432c22.1,0,40-17.9,40-40V185C369,162.9,351.1,145,329,145z M43.1,518.7c-6.2,6.2-14.7,9.9-24.1,9.9c-9.4,0-17.8-3.8-24-9.9c-6.2-6.2-10-14.6-10-23.9c0-9.4,3.8-17.8,10-24s14.6-10,24-10 c9.4,0,17.9,3.8,24,10c6.2,6.2,10,14.6,10,24C53,504.2,49.2,512.6,43.1,518.7z M104.8,529c-0.1-32.1-12.5-62.3-35.1-84.9 c-22.6-22.6-52.8-35.2-84.7-35.2V360c46.6,0,88.7,19,119.3,49.6c30.6,30.6,49.5,72.8,49.6,119.4H104.8z M192,529 c-0.1-114.2-92.8-207.1-206.9-207.1V273c70.6,0,134.5,28.7,180.8,75.1c46.3,46.4,75,110.3,75.1,180.9H192z"/>
</svg>
</a>
</div>
<div id="footer-initials">
NSS
</div>
<div id="footer-nav">
<a href="mailto:nicholas.selby@renewvia.com">CONTACT</a>
<a href="index.html">HOME</a>
<a href="sitemap.html">SITEMAP</a>
</div>
</div>
<p id="copyright">
© 2018–2026 This work by <a href="https://rupumped.github.io" property="cc:attributionName" rel="cc:attributionURL">Nicholas S. Selby</a> is licensed under a <a rel="license" target="_blank" href="http://creativecommons.org/licenses/by/4.0/">Creative Commons Attribution 4.0 International License</a>. Feel free to fork the <a rel="external" target="_blank" href="https://github.com/rupumped/rupumped.github.io">source code</a> from GitHub and create your own website using this template.
</p>
<!-- Tarpits -->
<div style="display:none;">
<a href="tarpit/entry.html">Archived Content</a>
</div>
<a href="tarpit/entry.html" style="opacity:0;position:absolute;top:0;left:0;width:1px;height:1px;">Archive</a>
<div style="position:absolute;left:-9999px;width:1px;height:1px;overflow:hidden;" aria-hidden="true">
<a href="tarpit/trap-1.html">Legacy Documentation</a>
<a href="tarpit/trap-2.html">Historical Posts</a>
<a href="tarpit/trap-3.html">Old Resources</a>
</div>
</footer>
</body>
</html>