Competing in the final!

Team Applied Robotics made it to the final of the Amazon Robotics Challenge 2017! We arrived in Nagoya the weekend before the competition and started running tests in simulation. This was the first chance most of us had to meet each other as development was split between team members in the Netherlands and Australia.

The Amazon Robotics Challenge involves two tasks, a stow task and a pick task. In the stow task, 32 items need to be placed from a tote into a storage system, simulating the packing of a storage system in a distribution centre. The items in the tote are piled together haphazardly, meaning recognition is difficult. The pick task involves moving specific items from a stocked storage system into three boxes, as if an order is being fulfilled for a customer. The final consists of a back to back stow and pick task. Before we arrived, we knew the stow task would be our weak point but that our storage system design, which uses multiple flat shelves to maximise surface area, would enable a good pick task.

On Thursday we had the official practice runs where teams could choose to practice a stow or pick task. We chose a stow task in preparation for the stow task on Friday. On Friday morning we arrived to find our Realsense cameras not working, due to an automatic kernel update that was downloaded the day before. After a stressful morning we worked out how to fix the issue and we had our stow run at 16.00, getting 5 points.

Immediately after the stow task we started testing the pick task ready for Saturday morning. We were the first team up at 9.30 so everything had to be in working order on Friday evening when the convention centre closed. Our pick run went very well and we got 140 points. This was better than expected, and opened the door to progressing to the finals. During the course of the day the other teams had their pick runs and we prepared in the event we made it through.  After the last team ran at 18.00 we found out we had made it into the top 8 teams competing in the final on Sunday, and we spent Saturday evening getting everything ready for our finals run on Sunday morning. In the finals, we stowed two items from the tote into our storage system and picked one item from the storage system into a box, giving us 20 points. At time of writing the other finalists are doing their final runs.

Team Applied Robotics 2017 pick task 

Team Applied Robotics returns to the Amazon Robotics Challenge 2017

We are excited to have been selected as one of sixteen teams worldwide to compete at the Amazon Robotics Challenge 2017! This year's event will be held at the end of July at Robocup 2017 in Nagoya, Japan.

The challenge has evolved through the years, and this year brings two main opportunities. In the previous editions the storage system was a standard shelf as used in Amazon warehouses, but this year teams can design the storage system themselves. This allows the storage system to be tuned to the robot, resulting in an integrated picking, stowing and storage system.

The second main change for this year is that only half of the competition set of items will be items from the training set. The other half will be new items that the teams get to see for 30 minutes immediately prior to the task attempts.

Our team for 2017 consists of members from the Netherlands and from Australia.

Competition days at the Amazon Picking Challenge 2016

Our first competition day started with a lot of work for our team. To compensate for inaccuracies in our object detection, we decided to implement force detection during pick-up and we adapted our pipeline to take an extra scan after picking up an item. This made picking of the items out of the bin a lot more robust, but we would lose a lot of time taking the extra scans of our objects.

 Our first challenge was the stow-task, where our robot had to pick up randomly placed items from a tote and place them in a shelf. After we mis-detected two objects, we decided to opt for a restart of the challenge. After the restart we did a lot better, we recognized one item based on our 2D-feature recognition algorithm and successfully placed it in the shelf. After that we also detected another item correctly, but ran out of time before we could stow it in the shelf. In the end we managed to end up in 12th place out of 16 teams.

For the second challenge, our robot had to perform the task of picking items from the shelf and placing them in the tote. Our challenge started a lot tougher, since all of the four items we could not pick up with our gripper were part of our assignment. We managed to detect, pick up and place two out of twelve items. However, one of these items was not picked up correctly, so we got some points deducted for that. In total, our team managed get to 12th place in this second challenge as well.

Busy first day in Leipzig

We've had a very busy first day setting up our system and getting it ready for the competition next friday. There were quite some issue with our camera calibration, but we were able to fix these with another lens.

In the meantime, Fabian has been 3D-printing some extra parts to make our cable guide more robust.

Tomorrow, we hope to fully test our pipeline with the new camera set-up so we'll be ready for the stowing task next friday!

Last preparations for Leipzig

With just a few days to go until the Amazon Picking Challenge in Leipzig, team Applied Robotics is getting ready for the challenge.

We have been upgrading our camera setup and our gripper. Also, last weeks we have been spending most of our spare time on improving our vision pipeline and implementing our tote-picking pipeline.

Even during our normal day jobs, our laptops have not been idle. They have been running tests to see if our motion planner returned collision-free paths.

Tuesday, we will depart for Leipzig and the competition will start on friday. Keep tuned for regular updates!

Team Applied Robotics wins Dutch derby in Denmark

During last week's RoboBusiness conference in Odense, Denmark, both Dutch teams exhibited their robot and prepared for the Amazon Picking Challenge by engaging in a friendly competition.

Team Applied Robotics arrived tuesday evening, the day before the conference started and we wasted no time setting up our system. Setting up our hardware was done in under 45 minutes, but our software was not yet fully tested and still needed quite some tweaking. Luckily wednesday was a relatively quiet day at the exhibition floor, so we could use our time testing the system and implementing a lot of improvements to our path planner.
Among other things, we are now able to publish collision objects and an octomap and our path planner is able to reliably plan the safest path in and out of the bins.

Friday was the day of our competition against Delft. We had agreed on a subset of items to be picked from the shelf and recruited an arbitrary judge to randomly place the items in the shelf. A large crowd had gathered to watch as the judge counted down until the start of our 10-minute match.

We were off to a quick start, and within 40 seconds we dropped our first item in the tote. Team Delft, however had some driver problems and had to opt for a restart. By the time Delft had restarted, we were already in a comfortable lead of 3 items. Picking up the rest of the items went smoothly, apart from the last one, which was placed in such a way our path planner could not find a collision-free path. Meanwhile, Delft was closing in on our score of 5 out of 6 items and deposited their third items with 2 minutes still on the clock. Luckily for us they did not manage to pick another item before the time ran out and therefore we won with 5 items against 3.

This challenge was a great opportunity for both teams to prepare for the real competition in Leipzig and we would like to thank team Delft for the great atmosphere and organization of this event.

Demonstration of Team Applied Robotics at RoboBusiness Europe

Team Applied Robotics was challenged by Team Delft to show off our robot at the RoboBusiness Europe Conference in Odense, Denmark. (1st of june untill the 3rd of june). Of course we were thrilled to take up this challenge, also because it is the best preparation we can get for the actual Amazon Picking Challenge in July.

Our two teams will be demonstrating our robot systems during the conference in a friendly competition. This will allow us to gauge the competition and fine-tune our strategy for the competition in Leipzig.

Read more about this event at:
http://www.robobusiness.eu/

We will keep you updated on the final preparations for this mini-competition in the next week!

Vacuum gripper re-design

For the new Amazon Picking Challenge we decided to improve our gripper design. The vacuum concept we used last year worked pretty well, but since this year the bins will be more crowded, our gripper needs to be even thinner.

Also for the bin-picking part of the challenge we could use an extra degree of freedom.

To make this new design, we've enlisted the help of last-year's mechanical engineer Maurice Ramaker and our latest addition to the team: Fabian Gouwens! We have developed a lot of different concepts and decided on two concepts for prototyping.

We have 3D printed the two concepts and put them to the test.

Team Applied Robotics to participate in APC 2016

On June 30th, 2016, the second installment of the Amazon Picking Challenge will take place as part of the Robocup event in Leipzig, Germany.

We, as Team Applied Robotics, are happy to announce that we have been selected to participate again in the Amazon Picking Challenge!

One of our team members from last year, John Gardenier, has left for a PhD position in Sydney and will not participate this year.
However, we are very happy to welcome Wim Abbeloos as a new team member! He joins Berend Kupers and Simon Jansen to form this year's core development team.

We' re very excited for the challenge this year, which includes not only picking items from the shelf. but also picking items from a bin and stowing them onto the shelf.

We've restarted development and will from now on regularly update this blog in the coming months.

Team Applied Robotics APC 2015 Summary

After participating in the inaugural Amazon Picking Challenge held alongside the IEEE ICRA 2015 conference in Seattle, we would like to describe our system in more detail. Our system was designed and implemented over a six-month period in our free time on a shoestring budget and delivered satisfactory results, resulting in a 10th place out of 28 participating teams from all over the world.

One of the first design decisions we made was to use a vacuum gripper as a mechanical solution was too large to be able to grab objects placed in the back of the shelf. An additional benefit was that compliant gripping was easy to achieve.

The robot arm we used was a UR5. Because we had access to the same robot for testing as that would be supplied at APC, we did not have to ship the actual robot. We did, however, need to ensure the robot base and gripper could be mounted and transported easily. We designed a base that could be taken apart completely and reassembled quickly. The gripper and camera bracket was also quickly mounted onto the end of the robot arm with just a couple of bolts. After assembly only two things needed to be calibrated: the position of the camera with respect to the robot arm and the position of the robot with respect to the cabinet. By defining a clear procedure for the set-up and calibration, the system was up and running within an hour.

The software architecture was based on ROS with separate modules for strategy, perception, motion planning and the interface to the UR controller. The motion planning used a heuristic approach to create Cartesian paths between the pick-up position (as determined by the perception module) and the drop-off position. For checking the Cartesian paths for collisions and reachability, URDF models of the robot and the shelf were used. After ensuring a collision-free path had been found, the robot controller executed the linear paths.

To acquire depth information, we used the Kinect V2 sensor mounted on the robot arm. This gave us sufficient resolution although issues with reflection due to the Time of Flight technique meant we needed careful filtering of the point clouds before processing. 

For distinguishing objects from the shelf, recognising them and determining their poses, we took a very pragmatic approach. Because the pose of cabinet was known, we could easily filter the required bin from the pointcloud. Then we used basic PCL functionality to segment the objects. Here, we assumed that objects were not touching. Although this was true for the majority of the bins, we knew that picking all objects would be unlikely. From each object segmented from the shelf we determined the dimensions and matched them against the target object. If there was a match, the grasp poses of the object were determined by approximating the object by a bounding box. Using this approach, we were able to correctly pick approximately 80% of the objects in realistic shelf set-ups.

Amazon Picking Challenge final results

On Thursday we had a meeting about the future of the Amazon Picking Challenge with representatives of all teams. We talked about how the results and code can be shared best, what can be done to make the challenge more interesting for spectators, how it can be organised next year and who would be interested in competing again.

In the evening Amazon hosted drinks and it was great to mingle and discuss approaches with other teams. The final standings can be found below. Team Applied Robotics placed 10th of 27 teams, with 11 points.

Conquering some of the most difficult items to pick

Competition report day 2

On Wednesday morning team MIT scored 88 points and team RBO from TUBerlin 148! There were some other interesting runs with a robotic humanoid hand and dual arm Yaskawa Motoman systems. 

Team RBO

Team RBO

At the end of Wednesday Team Applied Robotics was at shared 9th place. On Thursday morning the two final teams will have their runs. 

Competition report day 1

After arriving in Seattle, Team Applied Robotics wasted no time setting up our equipment. On Monday evening we could already set-up our frame and see the competition bays. Other teams were also busy setting up and we recognised quite a few of the robots from previous videos posted online.

The next morning, on Tuesday, we were back at the Washington State Convention Center ready to set-up for our official run at 14.00. The robot arm was stuck in traffic and only arrived on site 4 hours before our run, but luckily we were well prepared. Having practiced the entire set-up and calibration before, it took Simon and John little time to attach the gripper and camera, wire the vacuum cleaner motor and to calibrate the positions of the robot and the camera. We had plenty of time left over to test our entire system and we made some successful runs, picking items out of the cabinet. We were the only team picking multiple objects in quick succession. Soon a crowd had gathered and were filming our fast and smooth picking operation.

Around 13.00 we heard that the first team to compete had dropped out, so we would be the first team to compete in an hour. The judges arrived and confirmed the process. We got to choose a random usb stick containing our picking order and shelf layout. Ten minutes before time the judges filled the shelf. A crowd of about 50 had gathered around our bay and we started smoothly, picking the yellow duck dog toy in no time, landing us 11 points. The robot performed a wrist flip to get to the next bin in the bottom left hand corner. To grasp the second item, the glue, the robot rotated 90 degrees to pick it from the side. This resulted in the camera getting tangled in the vacuum cleaner hose and meant that upon moving into the bin the robot went into an emergency stop. We applied for our 5 minute reset period in which the shelf was restocked and we adjusted the hose cabling. After the reset we picked the first object successfully but unfortunately it got tangled again at the second object. This was the end of our official run.

Immediately afterwards, we took the opportunity to demonstrate our capabilities to the crowd. By slightly changing the orientation of the glue bottle we had a very successful run, scoring a hypothetical 67 points. Unfortunately for us, this run did not count.

During the rest of the afternoon we watched other challenges, talked with teams and gave a few interviews. By the end of the day we were still in third place. The main distinction between the teams was if we managed to pick one object or zero. No teams picked more than one item on the first day and most teams also used their reset time. While we had an unfortunate run, our system was capable of picking an entire shelf successfully, something the other teams on Tuesday didn't show. We got the comment that our system was "ready to be sold for production next week", compared to the last minute testing and hacking going on around us. About two thirds of the teams will have their runs on Wednesday.

Live stream is up

Team Applied Robotics is now broadcasting live from Bay 3! We are scheduled at 14.00 local time (23.00 Dutch time) and will be setting up until then 

https://youtu.be/vNb83194MHE

Off to Seattle

In the past week we completed our preparations and are very happy with the system performance. Everything is working as we designed it and integration went rather smoothly. This weekend we timed our set-up and packed up ready for Seattle.

We will be the 2nd team of more than 30 to compete. Our challenge time slot is on Tuesday 26th May in Bay 3 from 14.00 to 14.20 (23.00 - 23.20 NL time). We will try to have a live stream up from around 10.00 (19.00 NL time) if the convention centre wifi is fast enough. If so the link will be posted here.

We are really looking forward to seeing what the other teams have come up with and seeing how our design holds up to the competition!

Testing, Testing, Fine-tuning and some more testing

In the last two weeks, we finished integration of the complete system, including hardware as well as software.
After the successful initial tests, we started testing our set-up for the complete challenge cycle. From now on we will focus on fine-tuning the vision and path-planning algorithms, making only minor adjustments to make the system more robust.

Good start to last month of integration

In less than four weeks Team Applied Robotics will be competing in Seattle! In the past month we have been busy taking apart vacuum cleaners, designing a custom gripper and nozzle, integrating electronics, segmentation algorithms for the vision system, robot frame design and a custom motion coordinator.

This is a sketch of our frame to mount the robot, designed to be easy to disassemble and transport:

The gripping action is shown below:

Team Applied Robotics chosen for Amazon Picking Challenge Travel Reimbursement Award

Team Applied Robotics has been selected for a travel reimbursement award. 41 teams applied with videos of their progress so far. Our entry was chosen as one of the 25 most competitive submissions as can be found on the Amazon Picking Challenge news page.

The official Amazon Picking Challenge objects also arrived this week which will help immensely in improving our vision algorithms.

We look forward to the last two months leading to the Amazon Picking Challenge at ICRA 2015.

Improved vision algorithm and end effector design

Since our last update we have been working on further improvements to our vision algorithms and on a brand new end effector design.

The new end effector uses vacuum technology to be able to pick a wide variety of objects. The long and narrow design allows us to pick up objects from the back of the shelf and allows us to pick objects from the top and the side.

 The freshly arrived Kiva shelf comes in handy for testing these improvements.
 Please see the video below for our proof of concept tests for the vision and end effector design:

 

KIVA shelf

This week we had a special delivery from KIVA Systems in the US. A 64 kg package arrived with all the components to construct our own KIVA shelf to prepare for the Amazon Picking Challenge. It is a modular system and took about an hour to put together.

This type of shelf is used in highly automated Amazon warehouses. Note that the final picking task is still left to a human in the video below: