Engineering Community Portal

MERLOT Engineering

Welcome  – From the Editor

Welcome to the Engineering Portal on MERLOT. Here, you will find lots of resources on a wide variety of topics ranging from aerospace engineering to petroleum engineering to help you with your teaching and research.

As you scroll this page, you will find many Engineering resources.  This includes the most recently added Engineering material and members; journals and publications and Engineering education alerts and twitter feeds.  


Over 150 emeddable or downloadable 3D Simulations in the subject area of Automation, Electro/Mechanical, Process Control, and Renewable Energy.  Short 3-7 minute simulations that cover a range of engineering topics to help students understand conceptual engineering topics. 

Each video is hosted on Vimeo and can be played, embedded, or downloaded for use in the classroom or online.  Other option includes an embeddable HTML player created in Storyline with review questions for each simulation that reinforce the concepts learned. 

Made possible under a Department of Labor grant.  Extensive storyboard/ scripting work with instructors and industry experts to ensure content is accurate and up to date. 

Engineering Technology 3D Simulations in MERLOT

New Materials

New Members

Engineering on the Web

  • How Simulating for PCB Manufacturing Drives Profitability
    Sep 27, 2021 11:11 AM PDT
    Smart connected devices are everywhere, in homes, in transportation, and at work. This means electronic system design (ESD) is having a greater influence on almost every type of product requiring new simulation tools to help achieve electronic, electrical, mechanical, thermal, and connectivity goals. New printed circuit board development opportunities also bring new production challenges for manufacturers, who must ensure process efficiency starting with identifying potential defects and optimizing for manufacturing in the design phase.
  • How DeepMind Is Reinventing the Robot
    Sep 27, 2021 08:00 AM PDT
    Artificial intelligence has reached deep into our lives, though you might be hard pressed to point to obvious examples of it. Among countless other behind-the-scenes chores, neural networks power our virtual assistants, make online shopping recommendations, recognize people in our snapshots, scrutinize our banking transactions for evidence of fraud, transcribe our voice messages, and weed out hateful social-media postings. What these applications have in common is that they involve learning and operating in a constrained, predictable environment. But embedding AI more firmly into our endeavors and enterprises poses a great challenge. To get to the next level, researchers are trying to fuse AI and robotics to create an intelligence that can make decisions and control a physical body in the messy, unpredictable, and unforgiving real world. It's a potentially revolutionary objective that has caught the attention of some of the most powerful tech-research organizations on the planet. "I'd say that robotics as a field is probably 10 years behind where computer vision is," says Raia Hadsell, head of robotics at DeepMind, Google's London-based AI partner. (Both companies are subsidiaries of Alphabet.) This article is part of our special report on AI, “The Great AI Reckoning.” Even for Google, the challenges are daunting. Some are hard but straightforward: For most robotic applications, it's difficult to gather the huge data sets that have driven progress in other areas of AI. But some problems are more profound, and relate to longstanding conundrums in AI. Problems like, how do you learn a new task without forgetting the old one? And how do you create an AI that can apply the skills it learns for a new task to the tasks it has mastered before? Success would mean opening AI to new categories of application. Many of the things we most fervently want AI to do—drive cars and trucks, work in nursing homes, clean up after disasters, perform basic household chores, build houses, sow, nurture, and harvest crops—could be accomplished only by robots that are much more sophisticated and versatile than the ones we have now. Beyond opening up potentially enormous markets, the work bears directly on matters of profound importance not just for robotics but for all AI research, and indeed for our understanding of our own intelligence. Let's start with the prosaic problem first. A neural network is only as good as the quality and quantity of the data used to train it. The availability of enormous data sets has been key to the recent successes in AI: Image-recognition software is trained on millions of labeled images. AlphaGo, which beat a grandmaster at the ancient board game of Go, was trained on a data set of hundreds of thousands of human games, and on the millions of games it played against itself in simulation. To train a robot, though, such huge data sets are unavailable. "This is a problem," notes Hadsell. You can simulate thousands of games of Go in a few minutes, run in parallel on hundreds of CPUs. But if it takes 3 seconds for a robot to pick up a cup, then you can only do it 20 times per minute per robot. What's more, if your image-recognition system gets the first million images wrong, it might not matter much. But if your bipedal robot falls over the first 1,000 times it tries to walk, then you'll have a badly dented robot, if not worse. The problem of real-world data is—at least for now—insurmountable. But that's not stopping DeepMind from gathering all it can, with robots constantly whirring in its labs. And across the field, robotics researchers are trying to get around this paucity of data with a technique called sim-to-real. The San Francisco-based lab OpenAI recently exploited this strategy in training a robot hand to solve a Rubik's Cube. The researchers built a virtual environment containing a cube and a virtual model of the robot hand, and trained the AI that would run the hand in the simulation. Then they installed the AI in the real robot hand, and gave it a real Rubik's Cube. Their sim-to-real program enabled the physical robot to solve the physical puzzle. Despite such successes, the technique has major limitations, Hadsell says, noting that AI researcher and roboticist Rodney Brooks "likes to say that simulation is 'doomed to succeed.' " The trouble is that simulations are too perfect, too removed from the complexities of the real world. "Imagine two robot hands in simulation, trying to put a cellphone together," Hadsell says. If you allow them to try millions of times, they might eventually discover that by throwing all the pieces up in the air with exactly the right amount of force, with exactly the right amount of spin, that they can build the cellphone in a few seconds: The pieces fall down into place precisely where the robot wants them, making a phone. That might work in the perfectly predictable environment of a simulation, but it could never work in complex, messy reality. For now, researchers have to settle for these imperfect simulacrums. "You can add noise and randomness artificially," Hadsell explains, "but no contemporary simulation is good enough to truly recreate even a small slice of reality." Catastrophic forgetting: When an AI learns a new task, it has an unfortunate tendency to forget all the old ones. There are more profound problems. The one that Hadsell is most interested in is that of catastrophic forgetting: When an AI learns a new task, it has an unfortunate tendency to forget all the old ones. The problem isn't lack of data storage. It's something inherent in how most modern AIs learn. Deep learning, the most common category of artificial intelligence today, is based on neural networks that use neuronlike computational nodes, arranged in layers, that are linked together by synapselike connections. Before it can perform a task, such as classifying an image as that of either a cat or a dog, the neural network must be trained. The first layer of nodes receives an input image of either a cat or a dog. The nodes detect various features of the image and either fire or stay quiet, passing these inputs on to a second layer of nodes. Each node in each layer will fire if the input from the layer before is high enough. There can be many such layers, and at the end, the last layer will render a verdict: "cat" or "dog." Each connection has a different "weight." For example, node A and node B might both feed their output to node C. Depending on their signals, C may then fire, or not. However, the A-C connection may have a weight of 3, and the B-C connection a weight of 5. In this case, B has greater influence over C. To give an implausibly oversimplified example, A might fire if the creature in the image has sharp teeth, while B might fire if the creature has a long snout. Since the length of the snout is more helpful than the sharpness of the teeth in distinguishing dogs from cats, C pays more attention to B than it does to A. Each node has a threshold over which it will fire, sending a signal to its own downstream connections. Let's say C has a threshold of 7. Then if only A fires, it will stay quiet; if only B fires, it will stay quiet; but if A and B fire together, their signals to C will add up to 8, and C will fire, affecting the next layer. What does all this have to do with training? Any learning scheme must be able to distinguish between correct and incorrect responses and improve itself accordingly. If a neural network is shown a picture of a dog, and it outputs "dog," then the connections that fired will be strengthened; those that did not will be weakened. If it incorrectly outputs "cat," then the reverse happens: The connections that fired will be weakened; those that did not will be strengthened. Training of a neural network to distinguish whether a photograph is of a cat or a dog uses a portion of the nodes and connections in the network [shown in red, at left]. Using a technique called elastic weight consolidation, the network can then be trained on a different task, distinguishing images of cars from buses. The key connections from the original task are “frozen" and new connections are established [blue, at right]. A small fraction of the frozen connections, which would otherwise be used for the second task, are unavailable [purple, right diagram]. That slightly reduces performance on the second task. But imagine you take your dog-and-cat-classifying neural network, and now start training it to distinguish a bus from a car. All its previous training will be useless. Its outputs in response to vehicle images will be random at first. But as it is trained, it will reweight its connections and gradually become effective. It will eventually be able to classify buses and cars with great accuracy. At this point, though, if you show it a picture of a dog, all the nodes will have been reweighted, and it will have "forgotten" everything it learned previously. This is catastrophic forgetting, and it's a large part of the reason that programming neural networks with humanlike flexible intelligence is so difficult. "One of our classic examples was training an agent to play Pong," says Hadsell. You could get it playing so that it would win every game against the computer 20 to zero, she says; but if you perturb the weights just a little bit, such as by training it on Breakout or Pac-Man, "then the performance will—boop!—go off a cliff." Suddenly it will lose 20 to zero every time. This weakness poses a major stumbling block not only for machines built to succeed at several different tasks, but also for any AI systems that are meant to adapt to changing circumstances in the world around them, learning new strategies as necessary. There are ways around the problem. An obvious one is to simply silo off each skill. Train your neural network on one task, save its network's weights to its data storage, then train it on a new task, saving those weights elsewhere. Then the system need only recognize the type of challenge at the outset and apply the proper set of weights. But that strategy is limited. For one thing, it's not scalable. If you want to build a robot capable of accomplishing many tasks in a broad range of environments, you'd have to train it on every single one of them. And if the environment is unstructured, you won't even know ahead of time what some of those tasks will be. Another problem is that this strategy doesn't let the robot transfer the skills that it acquired solving task A over to task B. Such an ability to transfer knowledge is an important hallmark of human learning. Hadsell's preferred approach is something called "elastic weight consolidation." The gist is that, after learning a task, a neural network will assess which of the synapselike connections between the neuronlike nodes are the most important to that task, and it will partially freeze their weights. "There'll be a relatively small number," she says. "Say, 5 percent." Then you protect these weights, making them harder to change, while the other nodes can learn as usual. Now, when your Pong-playing AI learns to play Pac-Man, those neurons most relevant to Pong will stay mostly in place, and it will continue to do well enough on Pong. It might not keep winning by a score of 20 to zero, but possibly by 18 to 2. Raia Hadsell [top] leads a team of roboticists at DeepMind in London. At OpenAI, researchers used simulations to train a robot hand [above] to solve a Rubik's Cube.Top: DeepMind; Bottom: OpenAI There's an obvious side effect, however. Each time your neural network learns a task, more of its neurons will become inelastic. If Pong fixes some neurons, and Breakout fixes some more, "eventually, as your agent goes on learning Atari games, it's going to get more and more fixed, less and less plastic," Hadsell explains. This is roughly similar to human learning. When we're young, we're fantastic at learning new things. As we age, we get better at the things we have learned, but find it harder to learn new skills. "Babies start out having much denser connections that are much weaker," says Hadsell. "Over time, those connections become sparser but stronger. It allows you to have memories, but it also limits your learning." She speculates that something like this might help explain why very young children have no memories: "Our brain layout simply doesn't support it." In a very young child, "everything is being catastrophically forgotten all the time, because everything is connected and nothing is protected." The loss-of-elasticity problem is, Hadsell thinks, fixable. She has been working with the DeepMind team since 2018 on a technique called "progress and compress." It involves combining three relatively recent ideas in machine learning: progressive neural networks, knowledge distillation, and elastic weight consolidation, described above. Progressive neural networks are a straightforward way of avoiding catastrophic forgetting. Instead of having a single neural network that trains on one task and then another, you have one neural network that trains on a task—say, Breakout. Then, when it has finished training, it freezes its connections in place, moves that neural network into storage, and creates a new neural network to train on a new task—say, Pac-Man. Its knowledge of each of the earlier tasks is frozen in place, so cannot be forgotten. And when each new neural network is created, it brings over connections from the previous games it has trained on, so it can transfer skills forward from old tasks to new ones. But, Hadsell says, it has a problem: It can't transfer knowledge the other way, from new skills to old. "If I go back and play Breakout again, I haven't actually learned anything from this [new] game," she says. "There's no backwards transfer." That's where knowledge distillation, developed by the British-Canadian computer scientist Geoffrey Hinton, comes in. It involves taking many different neural networks trained on a task and compressing them into a single one, averaging their predictions. So, instead of having lots of neural networks, each trained on an individual game, you have just two: one that learns each new game, called the "active column," and one that contains all the learning from previous games, averaged out, called the "knowledge base." First the active column is trained on a new task—the "progress" phase—and then its connections are added to the knowledge base, and distilled—the "compress" phase. It helps to picture the two networks as, literally, two columns. Hadsell does, and draws them on the whiteboard for me as she talks. If you want to build a robot capable of accomplishing many tasks in a broad range of environments, you'd have to train it on every single one of them. The trouble is, by using knowledge distillation to lump the many individual neural networks of the progressive-neural-network system together, you've brought the problem of catastrophic forgetting back in. You'll change all the weights of the connections and render your earlier training useless. To deal with this, Hadsell adds in elastic weight consolidation: Each time the active column transfers its learning about a particular task to the knowledge base, it partially freezes the nodes most important to that particular task. By having two neural networks, Hadsell's system avoids the main problem with elastic weight consolidation, which is that all its connections will eventually freeze. The knowledge base can be as large as you like, so a few frozen nodes won't matter. But the active column itself can be much smaller, and smaller neural networks can learn faster and more efficiently than larger ones. So the progress-and-compress model, Hadsell says, will allow an AI system to transfer skills from old tasks to new ones, and from new tasks back to old ones, while never either catastrophically forgetting or becoming unable to learn anything new. Other researchers are using different strategies to attack the catastrophic forgetting problem; there are half a dozen or so avenues of research. Ted Senator, a program manager at the Defense Advanced Research Projects Agency (DARPA), leads a group that is using one of the most promising, a technique called internal replay. "It's modeled after theories of how the brain operates," Senator explains, "particularly the role of sleep in preserving memory." The theory is that the human brain replays the day's memories, both while awake and asleep: It reactivates its neurons in similar patterns to those that arose while it was having the corresponding experience. This reactivation helps stabilize the patterns, meaning that they are not overwritten so easily. Internal replay does something similar. In between learning tasks, the neural network recreates patterns of connections and weights, loosely mimicking the awake-sleep cycle of human neural activity. The technique has proven quite effective at avoiding catastrophic forgetting. There are many other hurdles to overcome in the quest to bring embodied AI safely into our daily lives. "We have made huge progress in symbolic, data-driven AI," says Thrishantha Nanayakkara, who works on robotics at Imperial College London. "But when it comes to contact, we fail miserably. We don't have a robot that we can trust to hold a hamster safely. We cannot trust a robot to be around an elderly person or a child." Nanayakkara points out that much of the "processing" that enables animals to deal with the world doesn't happen in the brain, but rather elsewhere in the body. For instance, the shape of the human ear canal works to separate out sound waves, essentially performing "the Fourier series in real time." Otherwise that processing would have to happen in the brain, at a cost of precious microseconds. "If, when you hear things, they're no longer there, then you're not embedded in the environment," he says. But most robots currently rely on CPUs to process all the inputs, a limitation that he believes will have to be surmounted before substantial progress can be made. You know the cat is never going to learn language, and I'm okay with that. His colleague Petar Kormushev says another problem is proprioception, the robot's sense of its own physicality. A robot's model of its own size and shape is programmed in directly by humans. The problem is that when it picks up a heavy object, it has no way of updating its self-image. When we pick up a hammer, we adjust our mental model of our body's shape and weight, which lets us use the hammer as an extension of our body. "It sounds ridiculous but they [robots] are not able to update their kinematic models," he says. Newborn babies, he notes, make random movements that give them feedback not only about the world but about their own bodies. He believes that some analogous technique would work for robots. At the University of Oxford, Ingmar Posner is working on a robot version of "metacognition." Human thought is often modeled as having two main "systems"—system 1, which responds quickly and intuitively, such as when we catch a ball or answer questions like "which of these two blocks is blue?," and system 2, which responds more slowly and with more effort. It comes into play when we learn a new task or answer a more difficult mathematical question. Posner has built functionally equivalent systems in AI. Robots, in his view, are consistently either overconfident or underconfident, and need ways of knowing when they don't know something. "There are things in our brain that check our responses about the world. There's a bit which says don't trust your intuitive response," he says. For most of these researchers, including Hadsell and her colleagues at DeepMind, the long-term goal is "general" intelligence. However, Hadsell's idea of an artificial general intelligence isn't the usual one—of an AI that can perform all the intellectual tasks that a human can, and more. Motivating her own work has "never been this idea of building a superintelligence," she says. "It's more: How do we come up with general methods to develop intelligence for solving particular problems?" Cat intelligence, for instance, is general in that it will never encounter some new problem that makes it freeze up or fail. "I find that level of animal intelligence, which involves incredible agility in the world, fusing different sensory modalities, really appealing. You know the cat is never going to learn language, and I'm okay with that." Hadsell wants to build algorithms and robots that will be able to learn and cope with a wide array of problems in a specific sphere. A robot intended to clean up after a nuclear mishap, for example, might have some quite high-level goal—"make this area safe"—and be able to divide that into smaller subgoals, such as finding the radioactive materials and safely removing them. I can't resist asking about consciousness. Some AI researchers, including Hadsell's DeepMind colleague Murray Shanahan, suspect that it will be impossible to build an embodied AI of real general intelligence without the machine having some sort of consciousness. Hadsell herself, though, despite a background in the philosophy of religion, has a robustly practical approach. "I have a fairly simplistic view of consciousness," she says. For her, consciousness means an ability to think outside the narrow moment of "now"—to use memory to access the past, and to use imagination to envision the future. We humans do this well. Other creatures, less so: Cats seem to have a smaller time horizon than we do, with less planning for the future. Bugs, less still. She is not keen to be drawn out on the hard problem of consciousness and other philosophical ideas. In fact, most roboticists seem to want to avoid it. Kormushev likens it to asking "Can submarines swim?...It's pointless to debate. As long as they do what I want, we don't have to torture ourselves with the question." Pushing a star-shaped peg into a star-shaped hole may seem simple, but it was a minor triumph for one of DeepMind's robots.DeepMind In the DeepMind robotics lab it's easy to see why that sort of question is not front and center. The robots' efforts to pick up blocks suggest we don't have to worry just yet about philosophical issues relating to artificial consciousness. Nevertheless, while walking around the lab, I find myself cheering one of them on. A red robotic arm is trying, jerkily, to pick up a star-shaped brick and then insert it into a star-shaped aperture, as a toddler might. On the second attempt, it gets the brick aligned and is on the verge of putting it in the slot. I find myself yelling "Come on, lad!," provoking a raised eyebrow from Hadsell. Then it successfully puts the brick in place. One task completed, at least. Now, it just needs to hang on to that strategy while learning to play Pong. This article appears in the October 2021 print issue as "How to Train an All-Purpose Robot."
  • Can Wearables "Testify" Against Their Owners?
    Sep 27, 2021 07:15 AM PDT
    Were he alive today, Edgar Allan Poe might be spinning tales such as "The Tell-Tale Fitbit." Of course, the (probably) imaginary beating of the murder victim's heart in Poe's classic tale of intolerable guilt was perceptible only to the narrator; but wearables recording heart rate, steps taken, and more are out in the world, real as can be—as is their data. Now courts are overwhelmingly deciding it might quantifiably aid in the quest to arrive at the truth. Two recent court decisions—one a civil case over an allegedly defective anatomical implant, the other a murder in rural Wisconsin—are the latest in a string of decisions confirming wearables data is fair game and can be pivotal in exposing a wrongdoing or exonerating an innocent person. As a rule of thumb, judges are making wearables data subject to nearly blanket discoverability—that is, any party to litigation needs to provide a stipulated amount of data to opposing counsel in preparing for trial if requested to do so. In the Missouri case Bartis v. Biomet, Inc., U.S. District Judge John Ross said the number of steps the plaintiff's Fitbit recorded was plainly relevant to his claims of severe injury due to a defective artificial hip despite his claims otherwise. Ross ordered that data to be turned over, while carefully stating that sleep, location, and heart rate data was not relevant and could be redacted. He also stopped short of issuing a decision that the steps data was prima facie accurate, saying such a decision was not called for at that point in the case. But that next step Ross alluded to, of introducing data's admissibility—its authoritatively attested validity—is another story, in which every case could rest on betting how much legal fact finders, be they judges or laypeople on a jury, are expected to know about how wearables work and how accurate their information is. The Wisconsin murder case State of Wisconsin v. Burch hinged in part on exactly that aspect; the victim's boyfriend, suspected of killing her and held for 17 days, was later exonerated in no small part because his Fitbit data showed he had moved only 12 steps during her final hours alive. The man who was ultimately convicted of the murder appealed a lower court's decision to the Wisconsin Supreme Court; the lower court decided wearables' widespread use, and an affidavit from Fitbit attesting the records the company supplied were accurate copies of the exonerated man's records, precluded the need for an expert attesting to the device's reliability. The higher court dismissed the appeal over the Fitbit data in a 6-1 vote, but experts in technology law say the dissenting judge made a strong argument and the near future holds a lot of uncertainty regarding what data will be allowed into testimony and what will be thrown out. "Technology may continue to develop to the point that people are so familiar with these devices and how they work that expert testimony won't be required in a lot of these cases, but I don't know if we are there yet," Michael Rabb, an associate attorney at Gallivan White Boyd in Greenville, South Carolina, who has written on trackers and the law, said. Megan Iorio, counsel at the Electronic Privacy Information Center (EPIC), concurred with Rabb. "When a piece of evidence is not self-explanatory or something a layman understands, then the procedure is that the party that introduces that evidence needs to have an expert come in and explain the technology," Iorio said. "When you just skip over that step and assume this is technology everybody uses, so everybody knows how it works, that is not a logical conclusion." Logical or not, a judge or jury weighing tracker data may not need to know the intricacies of how accelerometers measure steps or how a tracker's optical sensors measure heart rate, if—like the data that showed the Wisconsin murder victim's boyfriend was clearly stationary during the time she was killed—the amount by which a given amount of data may be skewed is not in question. If the story behind the data is foggier, though, a device's function and reliability may be the hinge upon which fate swings. "What happens when the data starts to tell a story that could be interpreted differently?" Ursula Gorham, senior lecturer at the University of Maryland School of Information Studies, said. "What happens then? What is it really telling us?" In the absence of slam-dunk decisions from higher courts setting clear precedent, EPIC's Iorio said it is likely that parties to any sort of case will seek previous decisions from varying jurisdictions that buttress their own claims: "Any case anywhere will help if there is no precedent from a higher court." Gallivan White Boyd's Rabb said the healthcare system may actually assist the legal system in establishing the reliability of wearables data as it gets more serious about evaluating it for accuracy and deploying trackers more widely in research and clinical use. "If we start to see the healthcare industry adopting tracking records in determining somebody's overall health, that will go a long way to helping courts determine the admissibility, whether this information is discoverable, or if an expert's going to be needed," he said. "If tracking information starts appearing in medical records, I think that's where the rubber hits the road in terms of saying 'OK, it's time to start letting this information in to be used one way or the other.'"
  • Enter Engineering, thyssenkrupp to tackle Tebinbulak iron ore deposit in Uzbekistan ...
    Sep 27, 2021 03:18 AM PDT
  • Parts obsolescence forces redesign and remanufacture of AN/ASK-7 data-transfer avionics ...
    Sep 27, 2021 02:33 AM PDT
  • Lamar University: Where multifaceted, job-ready engineers are made - Study International
    Sep 27, 2021 02:05 AM PDT
  • Evolution of Graphene Patterning: From Dimension Regulation to Molecular Engineering ...
    Sep 27, 2021 01:55 AM PDT
  • Military sets up simulation and design and development center to promote digital ...
    Sep 27, 2021 01:26 AM PDT
  • University Senate Names Gail Brion 2021 Outstanding Senator | UKNow
    Sep 27, 2021 01:14 AM PDT
  • Cambridge academics awarded fellowships at Royal Academy of Engineering | Varsity
    Sep 27, 2021 12:52 AM PDT
  • Patel Engineering, Manali Petrochemicals to Tour & Travel Stocks- here are top Buzzing ...
    Sep 27, 2021 12:51 AM PDT
  • Engineering graduates develop Deep Learning-based solution to detect phishing attacks - The Hindu
    Sep 27, 2021 12:41 AM PDT
  • The University of Maine was recently recognized amongst the top engineering and nursing ...
    Sep 27, 2021 12:07 AM PDT
  • Tetra Tech Nabs Slice of $800 Million Federal Contract | Los Angeles Business Journal
    Sep 26, 2021 11:56 PM PDT
  • ENGINEERING MOMENTUM - Board of Visitors
    Sep 26, 2021 11:21 PM PDT
  • SIA Engineering signs airframe maintenance agreement with Hawaiian Airlines | Singapore ...
    Sep 26, 2021 11:11 PM PDT
  • What are the evolving opportunities for the players in the Military Avionics Market?
    Sep 26, 2021 11:00 PM PDT
  • SEAS Appoints Committee on Plans to Consolidate Cambridge Buildings - The Harvard Crimson
    Sep 26, 2021 10:02 PM PDT
  • BOV Academic and Student Life Committee discusses advising, hears from new Engineering dean
    Sep 26, 2021 09:30 PM PDT
  • Student-led team to launch satellite in Spring 2022 - The Brown Daily Herald
    Sep 26, 2021 09:07 PM PDT
  • Why Lunar Engineering is an Exciting R&D Field for the “Hot Solar” - SolarPACES
    Sep 26, 2021 08:56 PM PDT
  • Will Fukushima’s Water Dump Set a Risky Precedent?
    Sep 24, 2021 12:23 PM PDT
    Since the Japanese earthquake and tsunami in March 2011, groundwater has been trickling through the damaged facilities at the Fukushima Daiichi nuclear power plant, filtering through the melted cores and fuel rods and becoming irradiated with a whole medley of radioisotopes. Japanese authorities have been pumping that water into a vast array of tanks on-site: currently over a thousand tanks, and adding around one new tank per week. Now, Japanese authorities are preparing to release that water into the Pacific Ocean. Even though they're treating and diluting the water first, the plan is meeting with vocal protests. From that opposition and from scientists' critiques of the process, the ongoing events at Fukushima leave an unprecedented example that other nuclear power facilities can watch and learn from. The release is slated to start in 2023, and potentially last for decades. This month, observers from the International Atomic Energy Agency have arrived in the country to inspect the process. And efforts are also underway to build an undersea tunnel that will discharge the water a kilometer away from the shore. Before they do that, they'll treat the water to cleanse it of radioactive contaminants. According to the authorities' account of the situation, there's one major contaminant that their system cannot cleanse: tritium. It's actually normal for nuclear power plants to release tritium into the air and water in their normal operations. In fact, pre-disaster, Fukushima Daiichi held boiling-water reactors, the lowest-tritium type of nuclear reactors. The Japanese government's solution is to dilute the tritium-contaminated water down to comparable levels. That's part of the reason the discharge will likely last several decades. "While one can argue whether such release limits are appropriate in general for normally operating facilities, the planned release, if carried out correctly, does not appear to be outside of the norm," says Edwin Lyman, director of nuclear power safety at the Union of Concerned Scientists. Even so, the plan has—perhaps expectedly—encountered some rather vocal opposition. Some of the loudest cries have come from within Japan, particularly from the fishing industry. Radiation levels in seafood from that coast are well within safety limits, but fishing cooperatives are concerned the plan is (once again) putting their reputations at stake. “Theoretically, it's possible to improve the situation a lot. In practice, they haven't done that.” Governments of neighboring countries, particularly China and South Korea, have also spoken out in opposition to the plan, claiming that the discharge will contaminate a shared ocean—even though, according to Lyman, both of these countries are already discharging highly diluted amounts of tritium from their own nuclear power facilities. The release plan has been floated for some years now, A plan to keep building more tanks indefinitely was considered, before being shelved: officially, due to lack of space. Furthermore, Lyman says that adding more tanks would only leave the site more exposed to natural disasters. That leaves the release. So, with that in mind, can the events at Fukushima offer other energy facilities around the world any lessons at all? For one, they're a good show of the need for emergency planning. "Every nuclear plant should be required to analyze the potential for such long-term consequences," says Lyman. "New nuclear plants, if built, should incorporate such evaluations into their siting decisions." But there's other things experts say that facilities could learn. For example, something that hasn't always been present in the Fukushima matter—working against it—has been transparency. Authorities at the plant haven't fully addressed the matter of non-tritium contaminants, according to Ken Buesseler, a marine radiochemist at Woods Hole Oceanographic Institution who has studied radioactivity in the ocean off Fukushima. Some contaminants—like caesium-137 and strontium-90—were present in the initial disaster in 2011. Others—like cobalt-60 and cerium-144—entered the water later. It isn't something the authorities have completely ignored. "Japan plans to run the water again through the decontamination process before release, and the dilution will further reduce the concentrations of the remaining isotopes," says Lyman. But Buesseler isn't convinced that it will be enough. "Theoretically, it's possible to improve the situation a lot," he says. "In practice, they haven't done that." Japanese authorities insist they can do so, but their ability, he says, hasn't been independently verified and peer-reviewed. So Buesseler says that more accountability would help. He wants to see the aforementioned independent verification happen, and he wants to see more acknowledgement of those non-tritium isotopes. And he wants the authorities to be more cognizant of the precedent they could be setting. "I'd hate to see every country that has radioactive waste start dumping waste into the ocean," he says. "It's a transboundary issue, in a way. It's something bigger than Japan, and something different from regular operation. I think they need to be at least open about that, getting international approval." Here, Lyman agrees. "This situation is unique and the decision to release the water into the sea should not set a precedent for any other project." But even taking all of that into account, some believe that, if anything, this is an example of a time when there simply is no choice but to take drastic action. "I believe that this action is necessary to avoid potentially worse consequences," says Lyman.
  • Deep Learning’s Diminishing Returns
    Sep 24, 2021 11:11 AM PDT
    Deep learning is now being used to translate between languages, predict how proteins fold, analyze medical scans, and play games as complex as Go, to name just a few applications of a technique that is now becoming pervasive. Success in those and other realms has brought this machine-learning technique from obscurity in the early 2000s to dominance today. Although deep learning's rise to fame is relatively recent, its origins are not. In 1958, back when mainframe computers filled rooms and ran on vacuum tubes, knowledge of the interconnections between neurons in the brain inspired Frank Rosenblatt at Cornell to design the first artificial neural network, which he presciently described as a "pattern-recognizing device." But Rosenblatt's ambitions outpaced the capabilities of his era—and he knew it. Even his inaugural paper was forced to acknowledge the voracious appetite of neural networks for computational power, bemoaning that "as the number of connections in the network increases...the burden on a conventional digital computer soon becomes excessive." This article is part of our special report on AI, “The Great AI Reckoning.” Fortunately for such artificial neural networks—later rechristened "deep learning" when they included extra layers of neurons—decades of Moore's Law and other improvements in computer hardware yielded a roughly 10-million-fold increase in the number of computations that a computer could do in a second. So when researchers returned to deep learning in the late 2000s, they wielded tools equal to the challenge. These more-powerful computers made it possible to construct networks with vastly more connections and neurons and hence greater ability to model complex phenomena. Researchers used that ability to break record after record as they applied deep learning to new tasks. While deep learning's rise may have been meteoric, its future may be bumpy. Like Rosenblatt before them, today's deep-learning researchers are nearing the frontier of what their tools can achieve. To understand why this will reshape machine learning, you must first understand why deep learning has been so successful and what it costs to keep it that way. Deep learning is a modern incarnation of the long-running trend in artificial intelligence that has been moving from streamlined systems based on expert knowledge toward flexible statistical models. Early AI systems were rule based, applying logic and expert knowledge to derive results. Later systems incorporated learning to set their adjustable parameters, but these were usually few in number. Today's neural networks also learn parameter values, but those parameters are part of such flexible computer models that—if they are big enough—they become universal function approximators, meaning they can fit any type of data. This unlimited flexibility is the reason why deep learning can be applied to so many different domains. The flexibility of neural networks comes from taking the many inputs to the model and having the network combine them in myriad ways. This means the outputs won't be the result of applying simple formulas but instead immensely complicated ones. For example, when the cutting-edge image-recognition system Noisy Student converts the pixel values of an image into probabilities for what the object in that image is, it does so using a network with 480 million parameters. The training to ascertain the values of such a large number of parameters is even more remarkable because it was done with only 1.2 million labeled images—which may understandably confuse those of us who remember from high school algebra that we are supposed to have more equations than unknowns. Breaking that rule turns out to be the key. Deep-learning models are overparameterized, which is to say they have more parameters than there are data points available for training. Classically, this would lead to overfitting, where the model not only learns general trends but also the random vagaries of the data it was trained on. Deep learning avoids this trap by initializing the parameters randomly and then iteratively adjusting sets of them to better fit the data using a method called stochastic gradient descent. Surprisingly, this procedure has been proven to ensure that the learned model generalizes well. The success of flexible deep-learning models can be seen in machine translation. For decades, software has been used to translate text from one language to another. Early approaches to this problem used rules designed by grammar experts. But as more textual data became available in specific languages, statistical approaches—ones that go by such esoteric names as maximum entropy, hidden Markov models, and conditional random fields—could be applied. Initially, the approaches that worked best for each language differed based on data availability and grammatical properties. For example, rule-based approaches to translating languages such as Urdu, Arabic, and Malay outperformed statistical ones—at first. Today, all these approaches have been outpaced by deep learning, which has proven itself superior almost everywhere it's applied. So the good news is that deep learning provides enormous flexibility. The bad news is that this flexibility comes at an enormous computational cost. This unfortunate reality has two parts. Extrapolating the gains of recent years might suggest that by 2025 the error level in the best deep-learning systems designed for recognizing objects in the ImageNet data set should be reduced to just 5 percent [top]. But the computing resources and energy required to train such a future system would be enormous, leading to the emission of as much carbon dioxide as New York City generates in one month [bottom]. SOURCE: N.C. THOMPSON, K. GREENEWALD, K. LEE, G.F. MANSO The first part is true of all statistical models: To improve performance by a factor of k, at least k2 more data points must be used to train the model. The second part of the computational cost comes explicitly from overparameterization. Once accounted for, this yields a total computational cost for improvement of at least k4. That little 4 in the exponent is very expensive: A 10-fold improvement, for example, would require at least a 10,000-fold increase in computation. To make the flexibility-computation trade-off more vivid, consider a scenario where you are trying to predict whether a patient's X-ray reveals cancer. Suppose further that the true answer can be found if you measure 100 details in the X-ray (often called variables or features). The challenge is that we don't know ahead of time which variables are important, and there could be a very large pool of candidate variables to consider. The expert-system approach to this problem would be to have people who are knowledgeable in radiology and oncology specify the variables they think are important, allowing the system to examine only those. The flexible-system approach is to test as many of the variables as possible and let the system figure out on its own which are important, requiring more data and incurring much higher computational costs in the process. Models for which experts have established the relevant variables are able to learn quickly what values work best for those variables, doing so with limited amounts of computation—which is why they were so popular early on. But their ability to learn stalls if an expert hasn't correctly specified all the variables that should be included in the model. In contrast, flexible models like deep learning are less efficient, taking vastly more computation to match the performance of expert models. But, with enough computation (and data), flexible models can outperform ones for which experts have attempted to specify the relevant variables. Clearly, you can get improved performance from deep learning if you use more computing power to build bigger models and train them with more data. But how expensive will this computational burden become? Will costs become sufficiently high that they hinder progress? To answer these questions in a concrete way, we recently gathered data from more than 1,000 research papers on deep learning, spanning the areas of image classification, object detection, question answering, named-entity recognition, and machine translation. Here, we will only discuss image classification in detail, but the lessons apply broadly. Over the years, reducing image-classification errors has come with an enormous expansion in computational burden. For example, in 2012 AlexNet, the model that first showed the power of training deep-learning systems on graphics processing units (GPUs), was trained for five to six days using two GPUs. By 2018, another model, NASNet-A, had cut the error rate of AlexNet in half, but it used more than 1,000 times as much computing to achieve this. Our analysis of this phenomenon also allowed us to compare what's actually happened with theoretical expectations. Theory tells us that computing needs to scale with at least the fourth power of the improvement in performance. In practice, the actual requirements have scaled with at least the ninth power. This ninth power means that to halve the error rate, you can expect to need more than 500 times the computational resources. That's a devastatingly high price. There may be a silver lining here, however. The gap between what's happened in practice and what theory predicts might mean that there are still undiscovered algorithmic improvements that could greatly improve the efficiency of deep learning. To halve the error rate, you can expect to need more than 500 times the computational resources. As we noted, Moore's Law and other hardware advances have provided massive increases in chip performance. Does this mean that the escalation in computing requirements doesn't matter? Unfortunately, no. Of the 1,000-fold difference in the computing used by AlexNet and NASNet-A, only a six-fold improvement came from better hardware; the rest came from using more processors or running them longer, incurring higher costs. Having estimated the computational cost-performance curve for image recognition, we can use it to estimate how much computation would be needed to reach even more impressive performance benchmarks in the future. For example, achieving a 5 percent error rate would require 10 19 billion floating-point operations. Important work by scholars at the University of Massachusetts Amherst allows us to understand the economic cost and carbon emissions implied by this computational burden. The answers are grim: Training such a model would cost US $100 billion and would produce as much carbon emissions as New York City does in a month. And if we estimate the computational burden of a 1 percent error rate, the results are considerably worse. Is extrapolating out so many orders of magnitude a reasonable thing to do? Yes and no. Certainly, it is important to understand that the predictions aren't precise, although with such eye-watering results, they don't need to be to convey the overall message of unsustainability. Extrapolating this way would be unreasonable if we assumed that researchers would follow this trajectory all the way to such an extreme outcome. We don't. Faced with skyrocketing costs, researchers will either have to come up with more efficient ways to solve these problems, or they will abandon working on these problems and progress will languish. On the other hand, extrapolating our results is not only reasonable but also important, because it conveys the magnitude of the challenge ahead. The leading edge of this problem is already becoming apparent. When Google subsidiary DeepMind trained its system to play Go, it was estimated to have cost $35 million. When DeepMind's researchers designed a system to play the StarCraft II video game, they purposefully didn't try multiple ways of architecting an important component, because the training cost would have been too high. At OpenAI, an important machine-learning think tank, researchers recently designed and trained a much-lauded deep-learning language system called GPT-3 at the cost of more than $4 million. Even though they made a mistake when they implemented the system, they didn't fix it, explaining simply in a supplement to their scholarly publication that "due to the cost of training, it wasn't feasible to retrain the model." Even businesses outside the tech industry are now starting to shy away from the computational expense of deep learning. A large European supermarket chain recently abandoned a deep-learning-based system that markedly improved its ability to predict which products would be purchased. The company executives dropped that attempt because they judged that the cost of training and running the system would be too high. Faced with rising economic and environmental costs, the deep-learning community will need to find ways to increase performance without causing computing demands to go through the roof. If they don't, progress will stagnate. But don't despair yet: Plenty is being done to address this challenge. One strategy is to use processors designed specifically to be efficient for deep-learning calculations. This approach was widely used over the last decade, as CPUs gave way to GPUs and, in some cases, field-programmable gate arrays and application-specific ICs (including Google's Tensor Processing Unit). Fundamentally, all of these approaches sacrifice the generality of the computing platform for the efficiency of increased specialization. But such specialization faces diminishing returns. So longer-term gains will require adopting wholly different hardware frameworks—perhaps hardware that is based on analog, neuromorphic, optical, or quantum systems. Thus far, however, these wholly different hardware frameworks have yet to have much impact. We must either adapt how we do deep learning or face a future of much slower progress. Another approach to reducing the computational burden focuses on generating neural networks that, when implemented, are smaller. This tactic lowers the cost each time you use them, but it often increases the training cost (what we've described so far in this article). Which of these costs matters most depends on the situation. For a widely used model, running costs are the biggest component of the total sum invested. For other models—for example, those that frequently need to be retrained— training costs may dominate. In either case, the total cost must be larger than just the training on its own. So if the training costs are too high, as we've shown, then the total costs will be, too. And that's the challenge with the various tactics that have been used to make implementation smaller: They don't reduce training costs enough. For example, one allows for training a large network but penalizes complexity during training. Another involves training a large network and then "prunes" away unimportant connections. Yet another finds as efficient an architecture as possible by optimizing across many models—something called neural-architecture search. While each of these techniques can offer significant benefits for implementation, the effects on training are muted—certainly not enough to address the concerns we see in our data. And in many cases they make the training costs higher. One up-and-coming technique that could reduce training costs goes by the name meta-learning. The idea is that the system learns on a variety of data and then can be applied in many areas. For example, rather than building separate systems to recognize dogs in images, cats in images, and cars in images, a single system could be trained on all of them and used multiple times. Unfortunately, recent work by Andrei Barbu of MIT has revealed how hard meta-learning can be. He and his coauthors showed that even small differences between the original data and where you want to use it can severely degrade performance. They demonstrated that current image-recognition systems depend heavily on things like whether the object is photographed at a particular angle or in a particular pose. So even the simple task of recognizing the same objects in different poses causes the accuracy of the system to be nearly halved. Benjamin Recht of the University of California, Berkeley, and others made this point even more starkly, showing that even with novel data sets purposely constructed to mimic the original training data, performance drops by more than 10 percent. If even small changes in data cause large performance drops, the data needed for a comprehensive meta-learning system might be enormous. So the great promise of meta-learning remains far from being realized. Another possible strategy to evade the computational limits of deep learning would be to move to other, perhaps as-yet-undiscovered or underappreciated types of machine learning. As we described, machine-learning systems constructed around the insight of experts can be much more computationally efficient, but their performance can't reach the same heights as deep-learning systems if those experts cannot distinguish all the contributing factors. Neuro-symbolic methods and other techniques are being developed to combine the power of expert knowledge and reasoning with the flexibility often found in neural networks. Like the situation that Rosenblatt faced at the dawn of neural networks, deep learning is today becoming constrained by the available computational tools. Faced with computational scaling that would be economically and environmentally ruinous, we must either adapt how we do deep learning or face a future of much slower progress. Clearly, adaptation is preferable. A clever breakthrough might find a way to make deep learning more efficient or computer hardware more powerful, which would allow us to continue to use these extraordinarily flexible models. If not, the pendulum will likely swing back toward relying more on experts to identify what needs to be learned. Special Report: The Great AI Reckoning READ NEXT: How the U.S. Army Is Turning Robots Into Team Players Or see the full report for more articles on the future of AI.
  • AMD’s Lisa Su Breaks Through the Silicon Ceiling
    Sep 24, 2021 11:00 AM PDT
    When Lisa Su became CEO of Advanced Micro Devices in 2014, the company was on the brink of bankruptcy. Since then, AMD's stock has soared—from less than US $2 per share to more than $110. The company is now a leader in high-performance computing. Su received accolades for spearheading AMD's turnaround, appearing on the Barron's Top CEOs of 2021 list, Fortune's 2020 Most Powerful Women, and CNN's Risk Takers. She recently added another honor: the IEEE Robert N. Noyce Medal. Su is the first woman to receive the award, which recognizes her "leadership in groundbreaking semiconductor products and successful business strategies that contributed to the strength of the microelectronics industry." Sponsored by Intel, the Noyce Medal is considered to be one of the semiconductor industry's most prestigious honors. "To be honest, I would have never imagined that I would receive the Noyce award," the IEEE Fellow says. "It's an honor of a lifetime. To have that recognition from my peers in the technical community is a humbling experience. But I love what I do and being able to contribute to the semiconductor industry." CLIMBING THE LEADERSHIP LADDER Su has long had a practical bent. She decided to study electrical engineering, she says, because she was drawn to the prospect of building hardware. "I felt like I was actually building and making things," she says. She attended MIT, where she earned bachelor's, master's, and doctoral degrees, all in EE, in 1990, 1991, and 1994. "It might surprise people that my parents would have preferred that I became a medical doctor," she says, laughing. "That was the most well-respected profession when I was growing up. But I never really liked the sight of blood. I ended up getting a Ph.D., which I guess was the next best thing." Her interest in semiconductors was sparked at MIT. As a doctoral candidate, Su was one of the first researchers to look into silicon-on-insulator (SOI) technology, according to an MIT Technology Review article about her. The then-unproven technique increased transistors' efficiency by building them atop layers of an insulating material. Today SOI is used either to boost the performance of microchips or to reduce their power requirements. Su has spent most of her career working on semiconductor projects for large companies. Along the way, she evolved from researcher to manager to top executive. Looking back, Lu divides her career path into two parts. The first 20 or so years she was involved in research and development; for the past 15 years, she has worked on the business side. Her first job was with Texas Instruments, in Dallas, where she was a member of the technical staff at the company's semiconductor process and device center. She joined in 1994, but after a year, she left for IBM, in New York. There, she was a staff member researching device physics. In 2000 she was assigned to be the technical assistant for IBM's chief executive. She later was promoted to director of emerging projects. She made the switch to management in 2006, when she was appointed vice president of IBM's semiconductor research and development center in New York. To better learn how to manage people, she took several leadership courses offered by the company. "I remember thinking after every class that I had learned something that I could apply going forward," she says. Su says she doesn't agree with the notion that leadership is an innate ability. "I really do believe that you can be trained to be a good leader," she says. "A lot of leadership isn't all that intuitive, but over time you develop an intuition for things to look for. Experience helps. "As engineers transition into business or management, you have to think about a different set of challenges that are not necessarily 'How do you make your transistor go faster?' but [instead] 'How do you motivate teams?' or 'How do you understand more about what customers want?' I've made my share of mistakes in those transitions, but I've also learned a lot. "I've also learned something from every boss I've ever worked for." "Great leaders can actually have their teams do 120 percent more than what they thought was possible." One of the first places she got a chance to put her training into action was at Freescale Semiconductor, in Austin, Texas. In 2007 she took over as chief technology officer and oversaw the company's research and development efforts. She was promoted to senior vice president and general manager of Freescale's networking and multimedia group. In that role, she was responsible for global strategy, marketing, and engineering for the embedded communications and applications processor business. She left in 2012 to join AMD, also in Austin, as senior vice president, overseeing the company's global business units. Two years later she was appointed president and CEO, the first woman to run a Fortune 500 semiconductor company. It took more than leadership skills to get to the top, she says. "It's a little bit of you have to be good [at what you do], but you also have to be lucky and be in the right place at the right time," she says. "I was fortunate in that I had a lot of opportunities throughout my career." As CEO, she fosters a supportive and diverse culture at AMD. "What I try to do is ensure that we're giving people a lot of opportunities," she says. "We have some very strong technical leaders at AMD who are women, so we're making progress. But of course it's nowhere near enough and it's nowhere near fast enough. There's always much more that can be done." Motivating employees is part of her job, she says. "One of the things I believe is that great leaders can actually have their teams do 120 percent more than what they thought was possible," she says. "What we try to do is to really inspire phenomenal and exceptional results." AMD's business is booming, and Su is credited with expanding the market for the company's chips beyond PCs to game consoles and embedded devices. AMD released products in 2017 with its Ryzen desktop processors and Epyc server processors for data centers. They are based on its Zen microarchitecture, which enabled the chips to quickly process more instructions than the competition. The Radeon line of graphics cards for gaming consoles debuted in 2000. The company's net income for last year was nearly $2.5 billion, according to Investor's Business Daily. WHAT'S AHEAD Today AMD is focused on building the next generation of supercomputers—which Su says will be "important in many aspects of research going forward." Last year the company announced its advanced CPUs, GPUs, and software will be powering Lawrence Livermore National Laboratory's El Capitan exascale-class supercomputer. Predicted to be the world's fastest when it goes into service in 2023, El Capitan is expected to expand the use of artificial intelligence and machine learning. There currently is a tightness in the semiconductor supply chain, Su acknowledges, but she says she doesn't think the shortage will fundamentally change what the company does in terms of technology or product development. "The way to think about semiconductor technology and road maps," she says, "is that the decisions about the products that we're building today were really decisions that were made three to five years ago. And the products or technical decisions that we're making today will affect our products three to five years down the road." The semiconductor industry has never been more interesting, she says, even with Moore's Law slowing down. Moore's Law, she says, "requires all of us to think differently about how we get to that next level of innovation. And it's not just about silicon innovation. It's also about packaging innovation, system software, and bringing together all those disciplines. There's a whole aspect to our work about just how to make our tools and our technologies easier to adopt." The COVID-19 pandemic has brought technology into the center of how people work, live, learn, and play, she notes. "Our goal," she says, "is to continue to make technology that touches more people's lives." Su was recently appointed to serve on the President's Council of Advisors on Science and Technology, a group of external advisers tasked with making science, technology, and innovation policy recommendations to the White House and President Biden. IMPORTANT ASSOCIATION Su joined IEEE while a student so she could access its technical content. "IEEE publications were just the most important," she says. "As a student, you wanted to publish in an IEEE journal or present at an IEEE conference. We all believed it was where people wanted to share their research. "I think IEEE is still the foremost organization for bringing researchers together to share their findings, to network, and to develop and build relationships," she says. "I've met many people through my IEEE connections, and they continue to be close colleagues. It's just a great organization to move the industry forward."
  • Video Friday: DARPA Subterranean Challenge Final
    Sep 24, 2021 08:00 AM PDT
    This week we have a special DARPA SubT edition of Video Friday, both because the SubT Final is happening this week and is amazing, and also because (if I'm being honest) the SubT Final is happening this week and is amazing and I've spent all week covering it mostly in a cave with zero access to Internet. Win-win, right? So today, videos to watch are DARPA's recaps of the preliminary competition days, plus (depending on when you're tuning in) a livestream of the prize round highlights, the awards ceremony, and the SubT Summit with roundtable discussions featuring both the Virtual and Systems track teams. DARPA Subterranean Challenge Final Event Day 1- Introduction to the SubT Challenge DARPA Subterranean Challenge Final Event - Day 2 - Competition Coverage DARPA Subterranean Challenge Final Event - Day 3 - Competition Coverage DARPA Subterranean Challenge Final Event Day 4 - Prize Round Coverage DARPA Subterranean Challenge Final Event Day 4 - Awards Ceremony and SubT Summit
  • Benchmark Shows AIs Are Getting Speedier
    Sep 24, 2021 06:00 AM PDT
    This week, AI industry group MLCommons released a new set of results for AI performance. The new list, MLPerf Version 1.1, follows the first official set of benchmarks by five months and includes more than 1800 results from 20 organizations, with 350 measurements of energy efficiency. The majority of systems improved by between 5-30 percent from earlier this year, with some more than doubling their previous performance stats, according to MLCommons. The new results come on the heels of the announcement, last week, of a new machine-learning benchmark, called TCP-AIx. In MLPerf's inferencing benchmarks, systems made up of combinations of CPUs and GPUs or other accelerator chips are tested on up to six neural networks performing a variety of common functions—image classification, object detection, speech recognition, 3D medical imaging, natural language processing, and recommendation. For commercially available datacenter-based systems they were tested under two conditions—a simulation of real datacenter activity where queries arrive in bursts and "offline" activity where all the data is available at once. Computers meant to work onsite instead of in the data center—what MLPerf calls the edge—were measured in the offline state and as if they were receiving a single stream of data, such as from a security camera. Although there were datacenter-class submissions from Dell, HPE, Inspur, Intel, LTech Korea, Lenovo, Nvidia, Neuchips, Qualcomm, and others, all but those from Qualcomm and Neuchips used Nvidia AI accelerator chips. Intel used no accelerator chip at all, instead demonstrating the performance of its CPUs alone. Neuchips only participated in the recommendation benchmark, as their accelerator, the RecAccel, is designed specifically to speed up recommender systems—which are used for recommending e-commerce items and for ranking search results. MLPerf tests six common AIs under several conditions.NVIDIA For the results Nvidia submitted itself, the company used software improvements alone to eke out as much as a 50 percent performance improvement over the past year. The systems tested were usually made up of one or two CPUs along with as many as eight accelerators. On a per-accelerator basis, systems with Nvidia A100 accelerators showed about double or more the performance those using the lower-power Nvidia A30. A30-based computers edged out systems based on Qualcomm's Cloud AI 100 in four of six tests in the server scenario. However, Qualcomm senior director of product management John Kehrli points out that his company's accelerators were deliberately limited to a datacenter-friendly 75-watt power envelope per chip, but in the offline image recognition task they still managed to speed past some Nvidia A100-based computers with accelerators that had peak thermal designs of 400 W each. Nvidia senior product manager for AI inferencing Dave Salvator pointed to two other outcomes for the company's accelerators: First, for the first time Nvidia A100 accelerators were paired with server-class Arm CPUs instead of x86 CPUs. The results were nearly identical between Arm and x86 systems across all six benchmarks. "That's an important milestone for Arm," says Salvator. "It's also a statement about the readiness of our software stack to be able to run the Arm architecture in a datacenter environment." Nvidia has made gains in AI using only software improvements.NVIDIA Separately from the formal MLPerf benchmarks, Nvidia showed off a new software technique called multi-instance GPU (MiG), which allows a single GPU to act as if it's seven separate chips from the point of view of software. When the company ran all six benchmarks simultaneously plus an extra instance of object detection (just as a flex, I assume) the results were 95 percent of the single-instance value. Nvidia A100-based systems also cleaned up on the edge server category, where systems are designed for places like stores and offices. These computers were tested along most of the same six benchmarks but with the recommender system swapped out for a low-res version of object detection. But in this category, there was a wider range of accelerators on offer, including Centaur's AI Integrated Coprocessor; Qualcomm's AI 100; Edgecortix' DNA-F200 v2, Nvidia's Jetson Xavier, and FuriosaAI's Warboy. Qualcomm topped the efficiency ranking for a machine vision test.Qualcomm With six tests under two conditions each in two commercial categories using systems that vary in number of CPUs and accelerators, MLPerf performance results don't really lend themselves to some kind of simple ordered list like achieves with supercomputing. The parts that come closest are the efficiency tests, which can be boiled down to inferences per second per watt for the offline component. Qualcomm systems were tested for efficiency on object recognition, object detection, and natural language processing in both the datacenter and edge categories. In terms of inferences per second per watt, they beat the Nvidia-backed systems at the machine vision tests, but not on language processing. Nvidia-accelerated systems took all the rest of the spots. In seeming opposition to MLPerf's multidimensional nature, a new benchmark was introduced last week that aims for a single number. The Transaction Processing Performance Council says the TCP-Aix benchmark: Generates and processes large volumes of data Trains preprocessed data to produce realistic machine learning models Conducts accurate insights for real-world customer scenarios based on the generated models Can scale to large distributed configurations Allows for flexibility in configuration changes to meet the demands of the dynamic AI landscape. The benchmark is meant to capture the complete end-to-end process of machine learning and AI, explains Hamesh Patel, chair of the TPCx-AI committee and principal engineer at Intel. That includes parts of the process that aren't included in MLPerf such as preparing the data and optimization. "There was no benchmark that emulates an entire data science pipeline," he says. "Customers have said it can take a week to prep [the data] and two days to train" a neural network. Big differences between MLPerf and TPC-Aix include the latter's dependence on synthetic data—data that resembles real data but is generated on the fly. MLPerf uses sets of real data for both training and inference, and MLCommons executive director David Kanter was skeptical about the value of results from synthetic data. Membership among MLCommons and TPC has a lot of overlap, so it remains to be seen which if either of the two benchmarks gains over the other in credibility. MLPerf certainly has the advantage for the moment, and computer system makers are already being asked for MLPerf data as part of requests for proposals, at least two MLPerf participants report.
  • DARPA SubT Finals: Robot Operator Wisdom
    Sep 23, 2021 12:09 PM PDT
    Each of the DARPA Subterranean Challenge teams is allowed to bring up to 20 people to the Louisville Mega Cavern for the final event. Of those 20 people, only five can accompany the robots to the course staging area to set up the robots. And of those five, just one person can be what DARPA calls the Human Supervisor. The Human Supervisor role, which most teams refer to as Robot Operator, is the only person allowed to interface with the robots while they're on the course. Or, it's probably more accurate to say that the team's base station computer is the only thing allowed to interface with robots on the course, and the human operator is the only person allowed to use the base station. The operator can talk to their teammates at the staging area, but that's about it—the rest of the team can't even look at the base station screens. Robot operator is a unique job that can be different for each team, depending on what kinds of robots that team has deployed, how autonomous those robots are, and what strategy the team is using during the competition. On the second day of the SubT preliminary competition, we talked with robot operators from all eight Systems Track teams to learn more about their robots, exactly what they do during the competition runs, and their approach to autonomy. "DARPA is interested in approaches that are highly autonomous without the need for substantive human interventions; capable of remotely mapping and/or navigating complex and dynamic terrain; and able to operate with degraded and unreliable communication links. The team is permitted to have a single Human Supervisor at a Base Station… The Human Supervisor is permitted to view, access, and/or analyze both course data and status data. Only the Human Supervisor is permitted to use wireless communications with the systems during the competition run." DARPA's idea here is that most of the robots competing in SubT will be mostly autonomous most of the time, hence their use of "supervisor" rather than "operator." Requiring substantial human-in-the-loop-ness is problematic for a couple of reasons—first, direct supervision requires constant communication, and we've seen how problematic communication can be on the SubT course. And second, operation means the need for a skilled and experienced operator, which is fine if you're a SubT team that's been practicing for years but could be impractical for a system of robots that's being deployed operationally. So how are teams making the robot operator role work, and how close are they to being robot supervisors instead? I went around the team garages on the second day of preliminary runs, and asked each team operator the same three questions about their roles. I also asked the operators, "What is one question I should I ask the next operator I talk to?" I added this as a bonus question, with each operator answering a question suggested by a different team operator. Team Robotika Robot Operator: Martin Dlouhy Tell me about the team of robots that you're operating and why you think it's the optimal team for exploring underground environments. This is the third time we've participated in a SubT event; we've tried various robots, small ones, bigger ones, but for us, these two robots seem to be optimal. Because we are flying from Czech Republic, the robots have to fit in our checked luggage. We also don't have the smaller robots or the drones that we had because like three weeks ago, we didn't even know if we would be allowed to enter the United States. So this is optimal for what we can bring to the competition, and we would like to demonstrate that we can do something with a simple solution. Once your team of robots is on the course, what do you do during the run? We have two robots, so it's easier than for some other teams. When the robots are in network range, I have some small tools to locally analyze data to help find artifacts that are hard for the robots to see, like the cellphone or the gas source. If everything goes fine, I basically don't have to be there. We've been more successful in the Virtual SubT competition because over half our team are software developers. We've really pushed hard to make the Virtual and System software as close as possible, and in Virtual, it's fully autonomous from beginning to end. There's one step that I do manually as operator—the robots have neural networks to recognize artifacts, but it's on me to click confirm to submit the artifact reports to DARPA. What autonomous decisions would you like your robots to be able to make that they aren't currently making, and what would it take to make that possible? I would actually like an operator-less solution, and we could run it, but it's still useful to have a human operator—it's safer for the robot, because it's obvious to a human when the robot is not doing well. Bonus operator question: What are the lowest and highest level decisions you have to make? The lowest level is, I open the code and change it on the fly. I did it yesterday to change some of the safety parameters. I do this all the time, it's normal. The highest level is asking the team, "guys, how are we going to run our robots today." Team MARBLE Robot Operator: Dan Riley Tell me about the team of robots that you're operating and why you think it's the optimal team for exploring underground environments. We've been using the Huskies [wheeled robots] since the beginning of the competition, it's a reliable platform with a lot of terrain capability. It's a workhorse that can do a lot of stuff. We were also using a tank-like robot at one time, but we had traversability issues so we decided to drop that one for this competition. We also had UAVs, because there's a lot of value in not having to worry about the ground while getting to areas that you can't get to with a ground robot, but unfortunately we had to drop that too because of the number of people and time that we had. We decided to focus on what we knew we could do well, and make sure that our baseline system was super solid. And we added the Spot robots within the last two months mostly to access areas that the Huskies can't, like going up and down stairs and tricky terrain. It's fast, and we really like it. Our team of robots is closely related to our deployment strategy. The way our planner and multi-robot coordination works is that the first robot really just plows through the course looking for big frontiers and new areas, and then subsequent robots will fill in the space behind looking for more detail. So we deploy the Spots first to push the environment since they're faster than the Huskies, and the Huskies will follow along and fill in the communications network. We know we don't want to run five robots tomorrow. Before we got here, we saw the huge cavern and thought that running more robots would be better. But based on the first couple runs, we now know that the space inside is much smaller, so we think four robots is good. Once your team of robots is on the course, what do you do during the run? The main thing I'm watching for is artifact reports from robots. While I'm waiting for artifact reports, I'm monitoring where the robots are going, and mainly I want to see them going to new areas. If I see them backtracking or going where another robot has explored already, I have the ability to send them new goal points in another area. When I get an artifact report, I look at the image to verify that it's a good report. For objects that may not be visible, like the cell phone [which has to be detected through the wireless signal it emits], if it's early in the mission I'll generally wait and see if I get any other reports from another robot on it. The localization isn't great on those artifacts, so once I do submit, if it doesn't score, I have to look around to find an area where it might be. For instance, we found this giant room with lots of shelves and stuff, and that's a great place to put a cell phone, and sure enough, that's where the cell phone was. What autonomous decisions would you like your robots to be able to make that they aren't currently making, and what would it take to make that possible? We pride ourselves on our autonomy. From the very beginning, that was our goal, and actually in earlier competitions I had very little control over the robot, I could not even send it a goal point. All I was getting was reports—it was a one-way street of information. I might have been able to stop the robot, but that was about it. Later on, we added the goal point capability and an option to drive the robot if I need to take over to get it out of a situation. I'm actually the lead for our Virtual Track team as well, and that's already decision-free. We're running the exact same software stack on our robots, and the only difference is that the virtual system also does artifact reporting. Honestly, I'd say that we're more effective having the human be able to make some decisions, but the exact same system works pretty well without having any human at all. Bonus operator question: How much sleep did you get last night? I got eight hours, and I could have had more, except I sat around watching TV for a while. We stressed ourselves out a lot during the first two competitions, and we had so many problems. It was horrible, so we said, "we're not doing that again!" A lot of our problems started with the setup and launching phase, just getting the robots started up and ready to go and out of the gate. So we spent a ton of time making sure that our startup procedures were all automated. And when you're able to start up easily, things just go well. Team Explorer Robot Operator: Chao Cao Tell me about the team of robots that you're operating and why you think it's the optimal team for exploring underground environments. We tried to diversify our robots for the different kinds of environments in the challenge. We have wheeled vehicles, aerial vehicles, and legged vehicles (Spot robots). Our wheeled vehicles are different sizes; two are relatively big and one is smaller, and two are articulated in the middle to give them better mobility performance in rough terrain. Our smaller drones can be launched from the bigger ground robots, and we have a larger drone with better battery life and more payload. In total, there are 11 robots, which is quite a lot to be managed by a single human operator under a constrained time limit, but if we manage those robots well, we can explore quite a large three dimensional area. Once your team of robots is on the course, what do you do during the run? Most of the time, to be honest, it's like playing a video game. It's about allocating resources to gain rewards (which in this case are artifacts) by getting the robots spread out to maximize coverage of the course. I'm monitoring the status of the robots, where they're at, and what they're doing. Most of the time I rely on the autonomy of the robots, including for exploration, coordination between multiple robots, and detecting artifacts. But there are still times when the robots might need my help, for example yesterday one of the bigger robots got itself stuck in the cave branch but I was able to intervene and get it to drive out. What autonomous decisions would you like your robots to be able to make that they aren't currently making, and what would it take to make that possible? Humans have a semantic understanding of the environment. Just by looking at a camera image, I can predict what an environment will be like and how risky it will be, but robots don't have that kind of higher level decision capability. So I might want a specific kind of robot to go into a specific kind of environment based on what I see, and I can redirect robots to go into areas that are a better fit for them. For me as an operator, at least from my personal experience, I think it's still quite challenging for robots to perform this kind of semantic understanding, and I still have to make those decisions. Bonus operator question: What is your flow for decision making? Before each run, we'll have a discussion among all the team members to figure out a rough game plan, including a deployment sequence—which robots go first, should the drones be launched from the ground vehicles or from the staging area. During the run, things are changing, and I have to make decisions based on the environment. I'll talk to the pit crew about what I can see through the base station, and then I'll make an initial proposal based on my instincts for what I think we should do. But I'm very focused during the run and have a lot of tasks to do, so my teammates will think about time constraints and how conservative we want to be and where other robots are because I can't think through all of those possibilities, and then they'll give me feedback. Usually this back and forth is quick and smooth. The Robot Operator is the only person allowed to interface with the robots while they're on the course—the operators pretty much controls the entire run by themselves.DARPA Team CTU-CRAS-NORLAB Robot Operator: Vojtech Salansky Tell me about the team of robots that you're operating and why you think it's the optimal team for exploring underground environments. We chose many different platforms. We have some tracked robots, wheeled robots, Spot robots, and some other experimental UGVs [small hexapods and one big hexapod], and every UGV has a different ability to traverse terrain, and we are trying to cover all possible locomotion types to be able to traverse anything on the course. Besides the UGVs, we're using UAVs as well that are able to go through both narrow corridors and bigger spaces. We brought a large number of robots, but the number that we're using, about ten, is enough to be able to explore a large part of the environment. Deploying more would be really hard for the pit crew of only five people, and there isn't enough space for more robots. Once your team of robots is on the course, what do you do during the run? It differs run by run, but the robots are mostly autonomous, so they decide where to go and I'm looking for artifact detections uploaded by the robots and approving or disapproving them. If I see that a robot is stuck somewhere, I can help it decide where to go. If it looks like a robot may lose communications, I can move some robots to make a chain from other robots to extend our network. I can do high level direction for exploration, but I don't have to—the robots are updating their maps and making decisions to best explore the whole environment. What autonomous decisions would you like your robots to be able to make that they aren't currently making, and what would it take to make that possible? Terrain assessment is subtle. At a higher level, the operator has to decide where to send a walking robot and where to send a rolling robot. It's tiny details on the ground and a feeling about the environment that help the operator make those decisions, and that is not done autonomously. Bonus operator question: How much bandwidth do you have? I'm on the edge. I have a map, I have some subsampled images, I have detections, I have topological maps, but it would be better to have everything in 4K and dense point clouds. Team CSIRO Data61 Robot Operator: Brendan Tidd Tell me about the team of robots that you're operating and why you think it's the optimal team for exploring underground environments. We've got three robot types that are here today—Spot legged robots, big tracked robots called Titans, and drones. The legged ones have been pretty amazing, especially for urban environments with narrow stairs and doorways. The tracked robots are really good in the tricky terrain of cave environments. And the drones can obviously add situational awareness from higher altitudes and detect those high artifacts. Once your team of robots is on the course, what do you do during the run? We use the term "operator" but I'm actually supervising. Our robots are all autonomous, they all know how to divide and conquer, they're all going to optimize exploring for depth, trying to split up where they can and not get in each other's way. In particular the Spots and the Titans have a special relationship where the Titan will give way to the Spot if they ever cross paths, for obvious reasons. So my role during the run is to coordinate node placement, that's something that we haven't automated—we've got a lot of information that comes back that I use to decide on good places to put nodes, and probably the next step is to automate that process. I also decide where to launch the drone. The launch itself is one click, but it still requires me to know where a good place is. If everything goes right, in general the robots will just do their thing. What autonomous decisions would you like your robots to be able to make that they aren't currently making, and what would it take to make that possible? The node drop thing is vital, but I think it's quite a complex thing to automate because there are so many different aspects to consider. The node mesh is very dynamic, it's affected by all the robots that are around it and obviously by the environment. Similarly, the drone launch, but that requires the robots to know when it's worth it to launch a drone. So those two things, but also pushing on the nav stack to make sure it can handle the crazy stuff. And I guess the other side is the detection. It's not a trivial thing knowing what's a false positive or not, that's a hard thing to automate. Bonus operator question: How stressed are you, knowing that it's just you controlling all the robots during the run? Coping with that is a thing! I've got music playing when I'm operating, I actually play in a metal band and we get on stage sometimes and the feeling is very similar, so it's really helpful to have the music there. But also the team, you know? I'm confident in our system, and if I wasn't, that would really affect my mental state. But we test a lot, and all that preparedness helps with the stress. Team CoSTAR Robot Operator: Kyohei Otsu Tell me about the team of robots that you're operating and why you think it's the optimal team for exploring underground environments. We have wheeled vehicles, legged vehicles, and aerial drones, so we can cover many terrains, handle stairs, and fly over obstacles. We picked three completely different mobility systems to be able to use many different strategies. The robots can autonomously adjust their roles by themselves; some explore, some help with communication for other robots. The number of robots we use depends on the environment—yesterday we deployed seven robots onto the course because we assumed that the environment would be huge, but it's a bit smaller than we expected, so we'll adapt our number to fit that environment. Once your team of robots is on the course, what do you do during the run? Our robots are autonomous, and I think we have very good autonomy software. During setup the robots need some operator attention; I have to make sure that everything is working including sensors, mobility systems, and all the algorithms. But after that, once I send the robot into the course, I totally forget about it and focus on another robot. Sometimes I intervene to better distribute our team of robots—that's something that a human is good at, using prior knowledge to understand the environment. And I look at artifact reports, that's most of my job. In the first phases of the Subterranean Challenge, we were getting low level information from the robots and sometimes using low level commands. But as the project proceeded and our technology matured, we found that it was too difficult for the operator, so we added functionality for the robot to make all of those low level decisions, and the operator just deals with high level decisions. What autonomous decisions would you like your robots to be able to make that they aren't currently making, and what would it take to make that possible? [answered by CoSTAR co-Team Lead Joel Burdick] Two things: the system reports that it thinks it found an artifact, and the operator has to confirm yes or no. He has to also confirm that the location seems right. The other thing is that our multi-robot coordination isn't as sophisticated as it could be, so the operator may have to retask robots to different areas. If we had another year, we'd be much closer to automating those things. Bonus Operator Question: Would you prefer if your system was completely autonomous and your job was not necessary? Yeah, I'd prefer that! Team Coordinated Robotics Robot Operator: Kevin Knoedler Tell me about the team of robots that you're operating and why you think it's the optimal team for exploring underground environments. The ideal mix in my mind is a fleet of small drones with lidar, but they are very hard to test, and very hard to get right. Ground vehicles aren't necessarily easier to get right, but they're easier to test, and if you can test something, you're a lot more likely to succeed. So that's really the big difference with the team of robots we have here. Once your team of robots is on the course, what do you do during the run? Some of the robots have an automatic search function where if they find something they report back, and what I'd like to be doing is just monitoring. But, the search function only works in larger areas. So right now the goal is for me to drive them through the narrow areas, get them into the wider areas, and let them go, but getting them to that search area is something that I mostly need to do manually one at a time. What autonomous decisions would you like your robots to be able to make that they aren't currently making, and what would it take to make that possible? Ideally, the robots would be able to get through those narrow areas on their own. It's actually a simpler problem to solve than larger areas, it's just not where we focused our effort. Bonus operator question: How many interfaces do you use to control your robots? We have one computer with two monitors, one controller, and that's it. Team CERBERUS Robot Operator: Marco Tranzatto Tell me about the team of robots that you're operating and why you think it's the optimal team for exploring underground environments. We have a mix of legged and flying robots, supported by a rover carrying a wireless antenna. The idea is to take legged robots for harsh environments where wheel robots may not perform as well, combined with aerial scouts that can explore the environment fast to provide initial situational awareness to the operator so that I can decide where to deploy the legged machines. So the goal is to combine the legged and flying robots in a unified mission to give as much information as possible to the human operator. We also had some bigger robots, but we found them to be a bit too big for the environment that DARPA has prepared for us, so we're not going to deploy them. Once your team of robots is on the course, what do you do during the run? We use two main modes: one is fully autonomous on the robots, and the other one is supervised autonomy where I have an overview of what the robots are doing and can override specific actions. Based on the high level information that I can see, I can decide to control a single robot to give it a manual waypoint to reposition it to a different frontier inside the environment. I can go from high level control down to giving these single commands, but the commands are still relatively high level, like "go here and explore." Each robot has artifact scoring capabilities, and all these artifact detections are sent to the base station once the robot is in communication range, and the human operator has to say, "okay this looks like a possible artifact so I accept it" and then can submit the position either as reported by the robot or the optimized position reported by the mapping server. What autonomous decisions would you like your robots to be able to make that they aren't currently making, and what would it take to make that possible? Each robot is autonomous by itself. But the cooperation between robots is still like… The operator has to set bounding boxes to tell each robot where to explore. The operator has a global overview, and then inside these boxes, the robots are autonomous. So I think at the moment in our pipeline, we still need a centralized human supervisor to say which robot explores in which direction. We are close to automating this, but we're not there yet. Bonus operator question: What is one thing you would add to make your life as an operator easier? I would like to have a more centralized way to give commands to the robots. At the moment I need to select each robot and give it a specific command. It would be very helpful to have a centralized map where I can tell a robot to say explore in a given area while considering data from a different robot. This was in our plan, but we didn't manage to deploy it yet.
  • COVID Breathalyzers Could Transform Rapid Testing
    Sep 23, 2021 12:00 PM PDT
    Concert venues, international airports and even restaurants are increasingly asking patrons for a recent negative COVID-19 test before entering their premises. Some organizations offer to test people on the spot as they enter. But current COVID-19 testing options aren't convenient enough for the kind of mass daily screening that some businesses would like to implement. Rapid antigen tests take about 15 minutes and are in short supply. Molecular test results—the gold standard—often take days to become available. Both typically require twirling a swab up the nose—not exactly something people like to do as they head for a cocktail. This has led many scientists to develop super-rapid testing methods using breath samples. Such devices are less socially awkward and can deliver results in under a minute—fast enough to feasibly screen large crowds as they pass through hubs. Here, IEEE Spectrum has selected five different approaches to analyzing breath for SARS-CoV-2, the virus that causes COVID-19. Some of these technologies can sense the virus directly. Others pick up indirect indicators, such as volatile organic compounds (VOCs). These molecules are present in healthy breath, but change in ratio when a person is infected with the virus. The technologies come from companies working in a wide range of applications that pivoted to COVID-19 when the pandemic hit. Steradian Technologies in the United States, for example, was building a product for human supersight, and managed to turn its optics technology into a diagnostic. No COVID-19 breathalyzer is widely available yet, but we might soon see them popping up in select settings globally. In May, Singapore provisionally approved a breath-based device from Breathonix and may use it to test travelers at a Singapore-Malaysia checkpoint, according to the company. A device from similarly named Breathomix, in the Netherlands, was recently used by a port company in Rotterdam to check about 3,500 employees daily. After more clinical validation, COVID-19 breath-based tests might finally give the world a more convenient and comfortable testing option. Photonics Biosensor Steradian Technologies, Houston Rumi Time to results: 30 seconds The user blows into a tube, and if the virus is present, its protein receptors will bind with a chemically reactive biosensor. The binding causes the biosensor to emit light in the form of photons. Mirrors inside the handheld device concentrate the light emissions to a single focal point, amplifying the signal and allowing a measurement to be made. Light emissions indicate a positive sample. About the size of: A glue gun Good for: Screening people before entering a business, concert, or school Electronic Nose Breathomix, Leiden, Netherlands SpiroNose Time to results: <1 minute A cylindrical electronic nose containing seven different biosensors detects exhaled VOCs in breath. The sensors, made of metal oxide semiconductors, react with compounds in the breath. The reactions cause measurable changes in the flow of electrons and indicate the presence of certain compounds. Pattern-recognition algorithms then compare the readings from the sample to those of healthy and infected sample profiles in its database. The device either delivers a negative result, or it recommends further testing. It does not, by itself, definitively provide a positive COVID-19 diagnosis. About the size of: A 500-milliliter water bottleGood for: Screening students in school or employees at large companies Mass Spectrometry Breathonix, Singapore BreFence Go COVID-19 Breath Test System Time to results: <1 minute In this "time-of-flight" mass spectrometry approach, VOCs in a breath sample are fragmented, given an electric charge and subjected to magnetic fields. This causes the fragments to take different trajectories depending on their mass-to-charge ratios. A detector records their abundance in a mass spectrum based on these ratios and the time it takes for the molecules to travel a known distance through the machine. The data helps identify the VOCs present in the sample. About the size of: A dishwasher Good for: hospitals, clinical laboratories, and point-of-care settings with trained operators Terahertz Spectroscopy RAM Group DE, Zweibrücken, Germany ThEA Terahertz Express Analyzer Time to results: 2 minutes A metamaterial nanoantenna deposited on glass creates resonances at specific points in the 1-to-2-terahertz range. These waves uniquely interact with SARS-CoV-2 and its protein structure. When the virus is present in a person's breath or throat-swab sample, it is drawn to the minuscule structures on the antenna. The presence of this viral matter generates disturbances in the resonances while interacting with the terahertz wave, creating a change in the spectrum. Detection of this signal indicates a positive result. About the size of: A large microwave ovenGood for: Screening people coming through transportation hubs and commercial centers Gas Chromatography—Ion Mobility Spectrometry Imspex Diagnostics, Abercynon, Wales BreathSpec Time to results: 8 minutes A breath sample moves through a gas-chromatography column, which separates particles based on their size. Molecules then enter an ion-mobility spectrometry chamber where they are ionized, accelerated across the chamber, and hit a Faraday plate. This results in a current that is specific to each molecule and is used to produce a 3D chromatogram that can be analyzed using machine learning algorithms. For COVID screening, the system looks for specific changes in the ratio of VOCs present in breath samples compared with those found in healthy breath. About the size of: A microwave ovenGood for: Testing travelers at airports, crowds at cultural festivals, staff at large companies This article appears in the October 2021 print issue as "Five COVID Breathalyzers."
  • How the U.S. Army Is Turning Robots Into Team Players
    Sep 23, 2021 08:39 AM PDT
    This article is part of our special report on AI, “The Great AI Reckoning.” "I should probably not be standing this close," I think to myself, as the robot slowly approaches a large tree branch on the floor in front of me. It's not the size of the branch that makes me nervous—it's that the robot is operating autonomously, and that while I know what it's supposed to do, I'm not entirely sure what it will do. If everything works the way the roboticists at the U.S. Army Research Laboratory (ARL) in Adelphi, Md., expect, the robot will identify the branch, grasp it, and drag it out of the way. These folks know what they're doing, but I've spent enough time around robots that I take a small step backwards anyway. The robot, named RoMan, for Robotic Manipulator, is about the size of a large lawn mower, with a tracked base that helps it handle most kinds of terrain. At the front, it has a squat torso equipped with cameras and depth sensors, as well as a pair of arms that were harvested from a prototype disaster-response robot originally developed at NASA's Jet Propulsion Laboratory for a DARPA robotics competition. RoMan's job today is roadway clearing, a multistep task that ARL wants the robot to complete as autonomously as possible. Instead of instructing the robot to grasp specific objects in specific ways and move them to specific places, the operators tell RoMan to "go clear a path." It's then up to the robot to make all the decisions necessary to achieve that objective. This article is part of our special report on AI, “The Great AI Reckoning.” The ability to make decisions autonomously is not just what makes robots useful, it's what makes robots robots. We value robots for their ability to sense what's going on around them, make decisions based on that information, and then take useful actions without our input. In the past, robotic decision making followed highly structured rules—if you sense this, then do that. In structured environments like factories, this works well enough. But in chaotic, unfamiliar, or poorly defined settings, reliance on rules makes robots notoriously bad at dealing with anything that could not be precisely predicted and planned for in advance. RoMan, along with many other robots including home vacuums, drones, and autonomous cars, handles the challenges of semistructured environments through artificial neural networks—a computing approach that loosely mimics the structure of neurons in biological brains. About a decade ago, artificial neural networks began to be applied to a wide variety of semistructured data that had previously been very difficult for computers running rules-based programming (generally referred to as symbolic reasoning) to interpret. Rather than recognizing specific data structures, an artificial neural network is able to recognize data patterns, identifying novel data that are similar (but not identical) to data that the network has encountered before. Indeed, part of the appeal of artificial neural networks is that they are trained by example, by letting the network ingest annotated data and learn its own system of pattern recognition. For neural networks with multiple layers of abstraction, this technique is called deep learning. Even though humans are typically involved in the training process, and even though artificial neural networks were inspired by the neural networks in human brains, the kind of pattern recognition a deep learning system does is fundamentally different from the way humans see the world. It's often nearly impossible to understand the relationship between the data input into the system and the interpretation of the data that the system outputs. And that difference—the "black box" opacity of deep learning—poses a potential problem for robots like RoMan and for the Army Research Lab. In chaotic, unfamiliar, or poorly defined settings, reliance on rules makes robots notoriously bad at dealing with anything that could not be precisely predicted and planned for in advance. This opacity means that robots that rely on deep learning have to be used carefully. A deep-learning system is good at recognizing patterns, but lacks the world understanding that a human typically uses to make decisions, which is why such systems do best when their applications are well defined and narrow in scope. "When you have well-structured inputs and outputs, and you can encapsulate your problem in that kind of relationship, I think deep learning does very well," says Tom Howard, who directs the University of Rochester's Robotics and Artificial Intelligence Laboratory and has developed natural-language interaction algorithms for RoMan and other ground robots. "The question when programming an intelligent robot is, at what practical size do those deep-learning building blocks exist?" Howard explains that when you apply deep learning to higher-level problems, the number of possible inputs becomes very large, and solving problems at that scale can be challenging. And the potential consequences of unexpected or unexplainable behavior are much more significant when that behavior is manifested through a 170-kilogram two-armed military robot. After a couple of minutes, RoMan hasn't moved—it's still sitting there, pondering the tree branch, arms poised like a praying mantis. For the last 10 years, the Army Research Lab's Robotics Collaborative Technology Alliance (RCTA) has been working with roboticists from Carnegie Mellon University, Florida State University, General Dynamics Land Systems, JPL, MIT, QinetiQ North America, University of Central Florida, the University of Pennsylvania, and other top research institutions to develop robot autonomy for use in future ground-combat vehicles. RoMan is one part of that process. The "go clear a path" task that RoMan is slowly thinking through is difficult for a robot because the task is so abstract. RoMan needs to identify objects that might be blocking the path, reason about the physical properties of those objects, figure out how to grasp them and what kind of manipulation technique might be best to apply (like pushing, pulling, or lifting), and then make it happen. That's a lot of steps and a lot of unknowns for a robot with a limited understanding of the world. This limited understanding is where the ARL robots begin to differ from other robots that rely on deep learning, says Ethan Stump, chief scientist of the AI for Maneuver and Mobility program at ARL. "The Army can be called upon to operate basically anywhere in the world. We do not have a mechanism for collecting data in all the different domains in which we might be operating. We may be deployed to some unknown forest on the other side of the world, but we'll be expected to perform just as well as we would in our own backyard," he says. Most deep-learning systems function reliably only within the domains and environments in which they've been trained. Even if the domain is something like "every drivable road in San Francisco," the robot will do fine, because that's a data set that has already been collected. But, Stump says, that's not an option for the military. If an Army deep-learning system doesn't perform well, they can't simply solve the problem by collecting more data. ARL's robots also need to have a broad awareness of what they're doing. "In a standard operations order for a mission, you have goals, constraints, a paragraph on the commander's intent—basically a narrative of the purpose of the mission—which provides contextual info that humans can interpret and gives them the structure for when they need to make decisions and when they need to improvise," Stump explains. In other words, RoMan may need to clear a path quickly, or it may need to clear a path quietly, depending on the mission's broader objectives. That's a big ask for even the most advanced robot. "I can't think of a deep-learning approach that can deal with this kind of information," Stump says. Robots at the Army Research Lab test autonomous navigation techniques in rough terrain [top, middle] with the goal of being able to keep up with their human teammates. ARL is also developing robots with manipulation capabilities [bottom] that can interact with objects so that humans don't have to.Evan Ackerman While I watch, RoMan is reset for a second try at branch removal. ARL's approach to autonomy is modular, where deep learning is combined with other techniques, and the robot is helping ARL figure out which tasks are appropriate for which techniques. At the moment, RoMan is testing two different ways of identifying objects from 3D sensor data: UPenn's approach is deep-learning-based, while Carnegie Mellon is using a method called perception through search, which relies on a more traditional database of 3D models. Perception through search works only if you know exactly which objects you're looking for in advance, but training is much faster since you need only a single model per object. It can also be more accurate when perception of the object is difficult—if the object is partially hidden or upside-down, for example. ARL is testing these strategies to determine which is the most versatile and effective, letting them run simultaneously and compete against each other. Perception is one of the things that deep learning tends to excel at. "The computer vision community has made crazy progress using deep learning for this stuff," says Maggie Wigness, a computer scientist at ARL. "We've had good success with some of these models that were trained in one environment generalizing to a new environment, and we intend to keep using deep learning for these sorts of tasks, because it's the state of the art." ARL's modular approach might combine several techniques in ways that leverage their particular strengths. For example, a perception system that uses deep-learning-based vision to classify terrain could work alongside an autonomous driving system based on an approach called inverse reinforcement learning, where the model can rapidly be created or refined by observations from human soldiers. Traditional reinforcement learning optimizes a solution based on established reward functions, and is often applied when you're not necessarily sure what optimal behavior looks like. This is less of a concern for the Army, which can generally assume that well-trained humans will be nearby to show a robot the right way to do things. "When we deploy these robots, things can change very quickly," Wigness says. "So we wanted a technique where we could have a soldier intervene, and with just a few examples from a user in the field, we can update the system if we need a new behavior." A deep-learning technique would require "a lot more data and time," she says. It's not just data-sparse problems and fast adaptation that deep learning struggles with. There are also questions of robustness, explainability, and safety. "These questions aren't unique to the military," says Stump, "but it's especially important when we're talking about systems that may incorporate lethality." To be clear, ARL is not currently working on lethal autonomous weapons systems, but the lab is helping to lay the groundwork for autonomous systems in the U.S. military more broadly, which means considering ways in which such systems may be used in the future. The requirements of a deep network are to a large extent misaligned with the requirements of an Army mission, and that's a problem. Safety is an obvious priority, and yet there isn't a clear way of making a deep-learning system verifiably safe, according to Stump. "Doing deep learning with safety constraints is a major research effort. It's hard to add those constraints into the system, because you don't know where the constraints already in the system came from. So when the mission changes, or the context changes, it's hard to deal with that. It's not even a data question; it's an architecture question." ARL's modular architecture, whether it's a perception module that uses deep learning or an autonomous driving module that uses inverse reinforcement learning or something else, can form parts of a broader autonomous system that incorporates the kinds of safety and adaptability that the military requires. Other modules in the system can operate at a higher level, using different techniques that are more verifiable or explainable and that can step in to protect the overall system from adverse unpredictable behaviors. "If other information comes in and changes what we need to do, there's a hierarchy there," Stump says. "It all happens in a rational way." Nicholas Roy, who leads the Robust Robotics Group at MIT and describes himself as "somewhat of a rabble-rouser" due to his skepticism of some of the claims made about the power of deep learning, agrees with the ARL roboticists that deep-learning approaches often can't handle the kinds of challenges that the Army has to be prepared for. "The Army is always entering new environments, and the adversary is always going to be trying to change the environment so that the training process the robots went through simply won't match what they're seeing," Roy says. "So the requirements of a deep network are to a large extent misaligned with the requirements of an Army mission, and that's a problem." Roy, who has worked on abstract reasoning for ground robots as part of the RCTA, emphasizes that deep learning is a useful technology when applied to problems with clear functional relationships, but when you start looking at abstract concepts, it's not clear whether deep learning is a viable approach. "I'm very interested in finding how neural networks and deep learning could be assembled in a way that supports higher-level reasoning," Roy says. "I think it comes down to the notion of combining multiple low-level neural networks to express higher level concepts, and I do not believe that we understand how to do that yet." Roy gives the example of using two separate neural networks, one to detect objects that are cars and the other to detect objects that are red. It's harder to combine those two networks into one larger network that detects red cars than it would be if you were using a symbolic reasoning system based on structured rules with logical relationships. "Lots of people are working on this, but I haven't seen a real success that drives abstract reasoning of this kind." For the foreseeable future, ARL is making sure that its autonomous systems are safe and robust by keeping humans around for both higher-level reasoning and occasional low-level advice. Humans might not be directly in the loop at all times, but the idea is that humans and robots are more effective when working together as a team. When the most recent phase of the Robotics Collaborative Technology Alliance program began in 2009, Stump says, "we'd already had many years of being in Iraq and Afghanistan, where robots were often used as tools. We've been trying to figure out what we can do to transition robots from tools to acting more as teammates within the squad." RoMan gets a little bit of help when a human supervisor points out a region of the branch where grasping might be most effective. The robot doesn't have any fundamental knowledge about what a tree branch actually is, and this lack of world knowledge (what we think of as common sense) is a fundamental problem with autonomous systems of all kinds. Having a human leverage our vast experience into a small amount of guidance can make RoMan's job much easier. And indeed, this time RoMan manages to successfully grasp the branch and noisily haul it across the room. Turning a robot into a good teammate can be difficult, because it can be tricky to find the right amount of autonomy. Too little and it would take most or all of the focus of one human to manage one robot, which may be appropriate in special situations like explosive-ordnance disposal but is otherwise not efficient. Too much autonomy and you'd start to have issues with trust, safety, and explainability. "I think the level that we're looking for here is for robots to operate on the level of working dogs," explains Stump. "They understand exactly what we need them to do in limited circumstances, they have a small amount of flexibility and creativity if they are faced with novel circumstances, but we don't expect them to do creative problem-solving. And if they need help, they fall back on us." RoMan is not likely to find itself out in the field on a mission anytime soon, even as part of a team with humans. It's very much a research platform. But the software being developed for RoMan and other robots at ARL, called Adaptive Planner Parameter Learning (APPL), will likely be used first in autonomous driving, and later in more complex robotic systems that could include mobile manipulators like RoMan. APPL combines different machine-learning techniques (including inverse reinforcement learning and deep learning) arranged hierarchically underneath classical autonomous navigation systems. That allows high-level goals and constraints to be applied on top of lower-level programming. Humans can use teleoperated demonstrations, corrective interventions, and evaluative feedback to help robots adjust to new environments, while the robots can use unsupervised reinforcement learning to adjust their behavior parameters on the fly. The result is an autonomy system that can enjoy many of the benefits of machine learning, while also providing the kind of safety and explainability that the Army needs. With APPL, a learning-based system like RoMan can operate in predictable ways even under uncertainty, falling back on human tuning or human demonstration if it ends up in an environment that's too different from what it trained on. It's tempting to look at the rapid progress of commercial and industrial autonomous systems (autonomous cars being just one example) and wonder why the Army seems to be somewhat behind the state of the art. But as Stump finds himself having to explain to Army generals, when it comes to autonomous systems, "there are lots of hard problems, but industry's hard problems are different from the Army's hard problems." The Army doesn't have the luxury of operating its robots in structured environments with lots of data, which is why ARL has put so much effort into APPL, and into maintaining a place for humans. Going forward, humans are likely to remain a key part of the autonomous framework that ARL is developing. "That's what we're trying to build with our robotics systems," Stump says. "That's our bumper sticker: 'From tools to teammates.' " This article appears in the October 2021 print issue as "Deep Learning Goes to Boot Camp." Special Report: The Great AI Reckoning READ NEXT: 7 Revealing Ways AIs Fail Or see the full report for more articles on the future of AI.
  • NYU Researchers Pave the Way for Future Shared Mobility
    Sep 23, 2021 07:43 AM PDT
    This article is sponsored by NYU Tandon School of Engineering. The collection of technologies and markets that comprise so-called "shared mobility" now constitutes a $60 billion market, according to some estimates. This enormous growth has at least in part been driven by the aim of reducing vehicle carbon emissions to address climate change concerns. In order for shared mobility to realize its aim of reducing pollution, there are a number of urban transportation elements that need to be taken into account, including car sharing services and micromobility offerings, such as e-bikes and scooters. As these shared mobility markets mature and develop, C2SMART, a U.S. Department of Transportation Tier 1 University Transportation Center at the NYU Tandon School of Engineering comprised of a consortium of universities, is leading research efforts to optimize these technologies to make them effective and efficient for our lives and our environments. Researchers at NYU have been at the forefront of much of C2SMART's contributions since first applying to be part of the U.S. federal government's University Transportation Centers program back in 2016. Since that time, the C2SMART Center at NYU has worked with shared mobility companies including, BMW ReachNow (now ShareNow), Lime and Via, and collaborated with automakers such as Ford Motor Company on some of its research. "The goal of the Center is primarily to tackle some of the most pressing issues that we see in mobility and in cities today and be able to come up with new solutions" "The goal of the Center is primarily to tackle some of the most pressing issues that we see in mobility and in cities today and be able to come up with new solutions," said Joseph Chow, Institute Associate Professor at NYU Tandon's Department of Civil and Urban Engineering and co-founding Deputy Director of C2SMART. Chow sees the Center as kind of bridge between local government agencies as well as the many private mobility companies and providers. "They come to us with problems and challenges that they see, and we try to come up with new solutions," added Chow. The agenda for the Center is driven in part by an annual Request for Proposal (RFP) process and partly from partnerships arranged with local agencies and private companies, according to Chow. As an example, C2SMART formed an on-call relationship with NYS DOT and for one of their tasks they are providing support in updating the New York 511 Rideshare program, which is a demand management system that has been in place for a few years now. "They want to update the program to next-generation technology that might consider more mobility services throughout the state to address equity needs as environmental needs," noted Chow. Joseph Chow, Institute Associate Professor at NYU Tandon's Department of Civil and Urban Engineering and co-founding Deputy Director of C2SMART.NYU Tandon Chow has focused some his recent research on developing the infrastructure for charging electric vehicles throughout New York City. While prior to this work there had been a lot of studies examining EV charging station locations, the charging considerations for mobility services tend to be trickier because they're on-demand, so their locations are not as known in advance. Chow and his colleagues addressed this problem by developing a look-ahead policy so that the model uses current data to anticipate where the future demand will be, taking into account the capacity at the different charging stations. This accounting for the capacity is a key difference from previous approaches. This new feature allows for the designing of different mixes of charger types and sizing of the charging stations. By being able to accommodate all these different sizing alternatives the model makes it possible to focus on different geographic locations, and changing the amount of capacity at these locations. Chow and his team collaborated with BMW ReachNow on this work, prior to the company merging with Car2Go. In another line of research, Chow has recently been working in the area of micromobility, a category of transit comprising e-scooters, mopeds, bicycles and the like, that has grown in popularity in cities around the world. In fact, Chow's recent research coincides with a current pilot e-scooter program in New York City. Acknowledging the rapid growth of e-scooter adoption, the NYU researchers looked specifically at the role that these vehicles play for the first and last mile in connecting travelers to public transit. The results support a relationship between how people use public transit and e-scooters, which bodes well for them reducing the number of cars in urban environments. NYU researchers discovered in their data analysis that one way that e-scooters will reduce cars is that fewer people will use carpooling from public transit hubs. The substitution of e-scooters for carpooling means that there might be less people dropping other people off because there will be e-scooters available. Taxis were also another mode of transport that looked to be displaced by e-scooters, albeit to a smaller extent. "This means that there'll be less need for vehicles, at least for short-distance trips," said Chow. "What we could witness in the long term might be a shift in the mode distribution by distance. For short-distance trips, you'll see e-scooters insert themselves into that spectrum." Micromobility, electric bikes, and EVs are among the topics tackled by researchers at the C2SMART Center at the NYU Tandon School of Engineering.Karl Philip Greenberg Chow recognizes that in the future there still will be a strong dependency on automobiles when used in conjunction with conventional fixed rail transit. However, he believes this might change if transit agencies consider running more on-demand services and micromobility. "In the long term, I think this would help to reduce the car and vehicle miles traveled, which would help to reduce congestion and greenhouse gas emissions," he added. While micromobility is a phenomenon currently taking hold in cities around the world, there's another emerging service called Mobility-as-a-Service (MaaS), in which a platform provider forms a single gateway to process multiple different options of trips into just different packages that people can purchase. This service is somewhat akin to how airlines have evolved, according to Chow. "Years ago, it was just individual airlines competing with each other," explained Chow. "But nowadays, when you book a trip, that single trip might be operated by three different airlines. As we head towards that in our public transit, I think there will be a bigger role to manage the demand through pricing and then try to make it more equitable for people." Along these lines, Chow highlighted the work his team has done with Italian start-up, NEXT Future Transportation, which offers modular, self-driving modules that look somewhat like a bus divided into smaller pods. This particular solution addresses one of the main problems in public transit of needing one or more transfers to get to your destination. This technology essentially allows passengers to transfer within the vehicle and then they can disband and go separate ways to be dropped off where they need to go. Chow's lab has studied methods to operate such systems as a transit service, including a recent award from the National Science Foundation. "Imagine a bracelet, and the different links in the bracelet are algorithmically programmed to go the last mile to a different neighborhood," explained Chow. "You just go to the link in the bracelet that'll take you to your neighborhood based on where you want to go, but you're going to go with other people who are going to that area."
  • China Aims for a Permanent Moon Base in the 2030s
    Sep 22, 2021 12:00 PM PDT
    On 3 January 2019, the Chinese spacecraft Chang'e-4 descended toward the moon. Countless craters came into view as the lander approached the surface, the fractal nature of the footage providing no sense of altitude. Su Yan, responsible for data reception for the landing at Miyun ground station, in Beijing, was waiting—nervously and in silence with her team—for vital signals indicating that optical, laser, and microwave sensors had combined effectively with rocket engines for a soft landing. "When the [spectral signals were] clearly visible, everyone cheered enthusiastically. Years of hard work had paid off in the most sweet way," Su recalls. Chang'e-4 had, with the help of a relay satellite out beyond the moon, made an unprecedented landing on the always-hidden lunar far side. China's space program, long trailing in the footsteps of the U.S. and Soviet (now Russian) programs, had registered an international first. The landing also prefigured grander Chinese lunar ambitions. In 2020 Chang'e-5, a complex sample-return mission, returned to Earth with young lunar rocks, completing China's three-step "orbit, land, and return" lunar program conceived in the early 2000s. These successes, together with renewed international scientific and commercial interest in the moon, have emboldened China to embark on a new lunar project that builds on the Chang'e program's newly acquired capabilities. The International Lunar Research Station (ILRS) is a complex, multiphase megaproject that the China National Space Administration (CNSA) unveiled jointly with Russia in June in St. Petersburg. Starting with robotic landing and orbiting missions in the 2020s, its designers envision a permanently inhabited lunar base by the mid-2030s. Objectives include science, exploration, technology verification, resource and commercial exploitation, astronomical observation, and more. ILRS will begin with a robotic reconnaissance phase running up to 2030, using orbiting and surface spacecraft to survey potential landing areas and resources, conduct technology-verification tests, and assess the prospects for an eventual permanent crewed base on the moon. The phase will consist of Chinese missions Chang'e-4, Chang'e-6 sample return, and the more ambitious Chang'e-7, as well as Russian Luna spacecraft, plus potential missions from international partners interested in joining the endeavor. Chang'e-7 will target a lunar south pole landing and consist of an orbiter, relay satellite, lander, and rover. It will also include a small spacecraft capable of "hopping" to explore shadowed craters for evidence of potential water ice, a resource that, if present, could be used in the future for both propulsion and supplies for astronauts. CNSA will help select the site for a two-stage construction phase that will involve in situ resource utilization (ISRU) tests with Chang'e-8, massive cargo delivery with precision landings, and the start of joint operations between partners. ISRU, in this case using the lunar regolith (the fine dust, soil, and rock that makes up most of the moon's surface) for construction and extraction of resources such as oxygen and water, would represent a big breakthrough. Being able to use resources already on the moon means fewer things need to be delivered, at great expense, from Earth. The China National Space Administration (CNSA) recently unveiled its plans for a lunar base in the 2030s, the International Lunar Research Station (ILRS). The first phase involves prototyping, exploration, and reconnaissance of possible ILRS locations.James Provost The utilization phase will begin in the early 2030s. It tentatively consists of missions numbered ILRS-1 through 5 and relies on heavy-lift launch vehicles to establish command, energy, and telecommunications infrastructure; experiment, scientific, and IRSU facilities; and Earth- and astronomical-observation capabilities. CNSA artist renderings indicate spacecraft will use the lunar regolith to make structures that would provide shielding from radiation while also exploring lava tubes as potential alternative areas for habitats. The completed ILRS would then host and support crewed missions to the moon in around 2036. This phase, CNSA says, will feature lunar research and exploration, technology verification, and expanding and maintaining modules as needed. These initial plans are vague, but senior figures in China's space industry have noted huge, if challenging, possibilities that could greatly contribute to development on Earth. Ouyang Ziyuan, a cosmochemist and early driving force for Chinese lunar exploration, notes in a July talk the potential extraction of helium-3, delivered to the lunar surface by unfiltered solar wind, for nuclear fusion (which would require major breakthroughs on Earth and in space). Another possibility is 3D printing of solar panels at the moon's equator, which would capture solar energy to be transmitted to Earth by lasers or microwaves. China is already conducting early research toward this end. As with NASA's Artemis plan, Ouyang notes that the moon is a stepping-stone to other destinations in the solar system, both through learning and as a launchpad. The more distant proposals currently appear beyond reach, but in its space endeavors China has demonstrated a willingness to develop capabilities and apply these for new possibilities. Sample-return tech from Chang'e-5 will next be used to collect material from a near-Earth asteroid around 2024. Near the end of the decade, this tech will contribute to the Tianwen-1 Mars mission's capabilities for an unprecedented Mars sample-return attempt. How the ILRS develops will then depend on success and science and resource findings of the early missions. China is already well placed to implement the early phases of the ILRS blueprint. The Long March 5, a heavy-lift rocket, had its first flight in 2016 and has since enabled the country to begin constructing a space station and to launch spacecraft such as a first independent interplanetary mission and Chang'e-5. To develop the rocket, China had to make breakthroughs in using cryogenic propellant and machining a new, wider-diameter rocket body. This won't be enough for larger missions, however. Huang Jun, a professor at Beihang University, in Beijing, says a super heavy-lift rocket, the high-thrust Long March 9, is a necessity for the future of Chinese aerospace. "Research and breakthroughs in key technologies are progressing smoothly, and the project may at any time enter the engineering-development stage." CNSA's plans for its international moon base involve a set of missions, dubbed ILRS-1 through ILRS-5, now projected between 2031 and 2035. IRLS-1, as planned, will in 2031 establish a command center and basic infrastructure. Subsequent missions over the ensuing four years would set up research facilities, sample collection systems, and Earth and spaceobservation capabilities.James Provost The roughly 100-meter-long, Saturn V–like Long March 9 will be capable of launching around 50 tonnes of payload to translunar injection. The project requires precision manufacturing of thin yet strong, 10-meter-diameter rocket stages and huge new engines. In Beijing, propulsion institutes under the China Aerospace Science and Technology Corp., recently produced an engineering prototype of a 220-tonne thrust staged-combustion liquid hydrogen/liquid oxygen engine. In a ravine near Xi'an, in north China, firing tests of a dual-chamber 500-tonne-thrust kerosene/liquid oxygen engine for the first stage have been carried out. Long March 9 is expected to have its first flight around 2030, which would come just in time to launch the robotic ILRS construction missions. A human-rated rocket is also under development, building on technologies from the Long March 5. It will feature similar but uprated versions of the YF-100 kerosene/liquid oxygen engine and use three rocket cores, in a similar fashion to SpaceX's Falcon Heavy. Its task will be sending a deep-space-capable crew spacecraft into lunar orbit, where it could dock with a lunar-landing stack launched by a Long March 9. The spacecraft itself is a new-generation advance on the Shenzhou, which currently ferries astronauts to and from low Earth orbit. A test launch in May 2020 verified that the new vessel can handle the greater heat of a higher-speed atmospheric reentry from higher, more energetic orbits. Work on a crew lander is also assumed to be underway. The Chang'e-5 mission was also seen as a scaled test run for human landings, as it followed a profile similar to NASA's Apollo missions. After lifting off from the moon, the ascent vehicle reunited and docked with a service module, much in the way that an Apollo ascent vehicle rejoined a command module in lunar orbit before the journey home. China and Russia are inviting all interested countries and partners to cooperate in the project. The initiative will be separate from the United States' Artemis moon program, however. The United States has long opposed cooperating with China in space, and recent geopolitical developments involving both Beijing and Moscow have made things worse still. As a result, China and Russia, its International Space Station partner, have looked to each other as off-world partners. "Ideally, we would have an international coalition of countries working on a lunar base, such as the Moon Village concept proposed by former ESA director-general Jan Wörner. But so far geopolitics have gotten in the way of doing that," says Brian Weeden, director of program planning for the Secure World Foundation. The final details and partners may change, but China, for its part, seems set on continuing the accumulation of expertise and technologies necessary to get to the moon and back, and stay there in the long term. This article appears in the October 2021 print issue as "China's Lunar Station Megaproject."
  • Air Quality: Easy to Measure, Tough to Fix
    Sep 22, 2021 08:00 AM PDT
    The summer of 2020 brought wildfire to Portland, Ore., as it did to so many other cities across the world. All outdoor activity in my neighborhood ceased for weeks, yet staying indoors didn't guarantee relief. The worst days left me woozy as my lone air purifier, whirring like a jet engine, failed to keep up. Obviously, the air in my home was bad. But I had no idea of how bad because, like most people, I had no way to measure it. That's changing, thanks to indoor air-quality monitors like Airthings' View Plus. Sold for US $299, the View Plus can gauge seven critical metrics: radon, particulates, carbon dioxide, humidity, temperature, volatile organic compounds, and air pressure. The monitor proved useful. I learned that cooking dinner can spike particulates into unhealthy territory for several hours, a sign that my oven vent is not working properly. The monitor also reported low levels of radon, proof that my home's radon mitigation system is doing its job. I had the monitor installed, working, and connected to the Airthings app less than 10 minutes after it arrived at my doorstep, in June. Reading the app was easy: It color-coded the results as good, fair, or poor. I have only one monitor, but the system can support multiple devices, making it possible to sniff out how air quality differs between rooms. You can also just move the device, though it needs time to update its readings. Airthings' monitor is unusual because it combines a radon sensor with other air-quality metrics, but it's certainly not alone. Alternatives are available from IQAir, Kaiterra,and Temtop, among others, and they range in price from $80 to $300. These monitors don't require permanent installation, so they're suitable for renters as well as owners. Of course, it's not enough to detect air pollutants; you must also remove them. That problem is more difficult. Ionization can itself create ozone. The state of California has banned such ozone generators entirely. Air purifiers surged in popularity through the second half of 2020 in response to dual airborne threats of COVID-19 and wildfire smoke. Companies responded to this demand at 2021's all-digital Consumer Electronics Show. LG led its presentation with personal air purifiers instead of televisions. Coway, Luft, and Scosche all showed new models, with Coway winning a CES Innovation Award for its new Design Flex purifiers. Unfortunately, consumers newly educated on indoor air quality will be puzzled about which air purifier, if any, is appropriate. Purifiers vary widely in the pollutants they claim to clean and how they claim to clean them. Most models advertise a HEPA air filter, which promises a specific standard of efficiency based on its rating, but this is often combined with unproven UV light, ionization, and ozone technologies that vaguely claim to catch toxins and kill pathogens, even COVID-19. This is the wild, wild west of air purification. It's true that an activated carbon filter can remove volatile organic compounds and ozone from the air. There's no common standard for efficiency, however, so shoppers must cross their fingers and hope for the best. Ionization, another popular feature, is no better. Studies suggest ionization can destroy viruses and bacteria in the air but, again, there's no common standard. In fact, ionization can itself create ozone. The state of California has banned such ozone generators entirely, but you'll still find these products on Amazon and other retailers. Studies even suggest the ionization feature in some purifiers may interact with the air in unpredictable ways, adding new pollutants. It's vital that companies designing air purifiers police their products and work together on standards that make sense to consumers. 2021's harsh fire season will keep demand high, but new, easy-to-use monitors like the Airthings View Plus will leave homeowners better informed about air quality—and ready to kick unproven purifiers to the curb. This article appears in the October 2021 print issue as "The Indoor Air-Quality Paradox."
  • Will This Jetpack Fly Itself?
    Sep 22, 2021 06:23 AM PDT
    Jetpacks might sound fun, but learning how to control a pair of jet engines strapped to your back is no easy feat. Now a British startup wants to simplify things by developing a jetpack with an autopilot system that makes operating it more like controlling a high-end drone than learning how to fly. Jetpacks made the leap from sci-fi to the real world as far back as the 1960s, but since then the they haven't found much use outside of gimmicky appearances in movies and halftime shows. In recent years though, the idea has received renewed interest. And its proponents are keen to show that the technology is no longer just for stuntmen and may even have practical applications. American firm Jetpack Aviation will teach anyone to fly its JB-10 jetpack for a cool $4,950 and recently sold its latest JB-12 model to an "undisclosed military." And an Iron Man-like, jet-powered flying suit developed by British start-up Gravity Industries has been tested as a way for marines to board ships and as a way to get medics to the top of mountains quickly. Flying jetpacks can take a lot of training to master though. That's what prompted Hollywood animatronics expert Matt Denton and Royal Navy Commander Antony Quinn to found Maverick Aviation, and develop one that takes the complexities of flight control out the pilot's hands. The Maverick Jetpack features four miniature jet turbines attached to an aluminum, titanium and carbon fiber frame, and will travel at up to 30 miles per hour. But the secret ingredient is software that automatically controls the engines to maintain a stable hover, and seamlessly convert the pilot's instructions into precise movements. "It's going to be very much like flying a drone," says Denton. "We wanted to come up with something that anyone could fly. It's all computer-controlled and you'll just be using the joystick." One of the key challenges, says Denton, was making the engines responsive enough to allow the rapid tweaks required for flight stabilization. This is relatively simple to achieve on a drone, whose electric motors can be adjusted in a blink of an eye, but jet turbines can take several seconds to ramp up and down between zero and full power. To get around this, the company added servos to each turbine that let them move independently to quickly alter the direction of thrust—a process known as thrust vectoring. By shifting the alignment of the four engines the flight control software can keep the jetpack perfectly positioned using feedback from inertial measurement units, GPS, altimeters and ground distance sensors. Simple directional instructions from the pilot can also be automatically translated into the required low-level tweaks to the turbines. It's a clever way to improve the mobility of the system, says Ben Akih-Kumgeh, an associate professor of aerospace engineering at Syracuse University. "It's not only a smart way of overcoming any lag that you may have, but it also helps with the lifespan of the engine," he adds. “[In] any mechanical system, the durability depends on how often you change the operating conditions." The software is fairly similar to a conventional drone flight controller, says Denton, but they have had to accommodate some additional complexities. Thrust magnitude and thrust direction have to be managed by separate control loops due to their very different reaction times, but they still need to sync up seamlessly to coordinate adjustments. The entire control process is also complicated by the fact that the jetpack has a human strapped to it. "Once you've got a shifting payload, like a person who's wobbling their arms around and moving their legs, then it does become a much more complex problem," says Denton. In the long run, says Denton, the company hopes to add higher-level functions that could allow the jetpack to move automatically between points marked on a map. The hope is that by automating as much of the flight control as possible, users will be able to focus on the task at hand, whether that's fixing a wind turbine or inspecting a construction site. Surrendering so much control to a computer might give some pause for thought, but Denton says there will be plenty of redundancy built in. "The idea will be that we'll have plenty of fallback modes where, if part of the system fails, it'll fall back to a more manual flight mode," he said. "The user would have training to basically tackle any of those conditions." It might be sometime before you can start basic training, though, as the company has yet to fly their turbine-powered jetpack. Currently, flight testing is being conducted on an scaled down model powered by electric ducted fans, says Denton, though their responsiveness has been deliberately dulled so they behave like turbines. The company is hoping to conduct the first human test flights next summer. Don't get your hopes up about commuting to work by jetpack any time soon though, says Akih-Kumgeh. The huge amount of noise these devices produce make it unlikely that they would be allowed to operate within city limits. The near term applications are more likely to be search and rescue missions where time and speed trump efficiency, he says.
  • DARPA SubT Final: How It Works and How to Watch
    Sep 21, 2021 01:22 PM PDT
    The preliminary rounds of the DARPA Subterranean Challenge Finals are kicking off today. It's been a little bit since the last DARPA SubT event—the Urban Circuit squeaked through right before the pandemic hit back in February of 2020, and the in-person Cave Circuit originally scheduled for later that year was canceled. So if it's been a while since you've thought about SubT, this article will provide a very brief refresher, and we'll also go through different ways in which you can follow along with the action over the course of the week. The overall idea of the DARPA Subterranean Challenge is to get teams of robots doing useful stuff in challenging underground environments. "Useful stuff" means finding important objects or stranded humans, and "challenging underground environments" includes human-made tunnel systems, the urban underground (basements, subways, etc), as well as natural caves. And "teams of robots" can include robots that drive, crawl, fly, walk, or anything in between. Over the past few years, teams of virtual and physical robots have competed in separate DARPA-designed courses representing each of those three underground domains. The Tunnel Event took place in an old coal mine, the Urban Event took place in an unfinished nuclear reactor complex, and the Cave Event—well, that got canceled because of COVID, but lots of teams found natural caves to practice in anyway. So far, we've learned that underground environments are super hard for robots. Communications are a huge problem, and robots have to rely heavily on autonomy and teamwork rather than having humans tell them what to do, although we've also seen all kinds of clever solutions to this problem. Mobility is tough, but legged robots have been surprisingly useful, and despite the exceptionally unfriendly environment, drones are playing a role in the challenge as well. Each team brings a different approach to the Subterranean Challenge, and every point scored represents progress towards robots that can actually be helpful in underground environments when we need them to be. The final Subterranean Challenge event, happening this week includes both a Virtual Track for teams competing with virtual robots, and a Systems Track for teams competing with physical robots. Let's take a look at how the final competition will work, and then the best ways to watch what's happening. How It Works If you've been following along with the previous circuits (Tunnel and Urban), the overall structure of the Final will be somewhat familiar, but there are some important differences to keep in mind. First, rather than being a specific kind of underground environment, the final course will incorporate elements from all three environments as well as some dynamic obstacles that could include things like closing doors or falling rocks. Only DARPA knows what the course looks like, and it will be reconfigured every day. Each of the Systems Track teams will have one 30-minute run on the course on Tuesday and another on Wednesday. 30 minutes is half the amount of time that teams have had in previous competitions. A Team's preliminary round score will be the sum of the scores of the two runs, but every team will get to compete in the final on Thursday no matter what their score is: the preliminary score only serves to set the team order, with higher scoring teams competing later in the final event. The final scoring run for all teams happens on Thursday. There will be one single 60 minute run for each team, which is a departure from previous events: if a team's robots misbehave on Thursday, that's just too bad, because there is no second chance. A team's score on the Thursday run is what will decide who wins the Final event; no matter how well a team did in previous events or in the preliminary runs this week, the Thursday run is the only one that counts for the prize money. Scoring works the same as in previous events. There will be artifacts placed throughout the course, made up of 10 different artifact types, like cell phones and fire extinguishers. Robots must identify the specific artifact type and transmit its location back to the starting area, and if that location is correct within 5 meters, a point is scored. Teams have a limited number of scoring attempts, though: there will be a total of 40 artifacts on the course for the prize round, but only 45 scoring attempts are allowed. And if a robot locates an artifact but doesn't manage to transmit that location back to base, it doesn't get that point. The winning team is the one with the most artifacts located in the shortest amount of time (time matters only in the event of a tie). The Virtual Track winners will take home $750k, while the top System Track team wins $2 million, with $1 million for second and $500k for third. If that's not enough of a background video for you, DARPA has helpfully provided this hour long video intro. How to Watch Watching the final event is sadly not as easy as it has been for previous events. Rather than publicly live streaming raw video feeds from cameras hidden inside the course, DARPA will instead record everything themselves and then produce edited and commentated video recaps that will post to YouTube the following day. So, Tuesday's preliminary round content will be posted on Wednesday, the Wednesday prelims post Thursday, and the Final event on Thursday will be broadcast on Friday as the teams themselves watch. Here's the schedule: The SubT Summit on Friday afternoon consists of roundtable discussions from both the Virtual Track teams and System Track teams; those will be from 2:30 to 3:30 and 4:00 to 5:00 respectively, with a half hour break in the middle. All of these streams are pre-scheduled on the DARPA YouTube channel. DARPA will also be posting daily blogs and sharing photos here. After the Thursday Final, it might be possible for us to figure out a likely winner based on artifact counts. But the idea is that even though the Friday broadcast is one day behind the competition, both we and the teams will be finding out what happened (and who won) at the same time—that's what will happen on the Friday livestream. Saturday, incidentally, has been set aside for teams to mess around on the course if they want to. This won't be recorded or broadcast at all, but I'll be there for a bit to see what happens. If you're specifically looking for a way to follow along in real time, I'm sorry to say that there isn't one. There will be real-time course feeds in the press room, but press is not allowed to share any of the things that we see. So if you're looking for details that are as close to live as possible, I'd recommend checking out Twitter, because many teams and team members are live Tweeting comments and pictures and stuff, and the easiest way to find that is by searching for the #SubTChallenge hashtag. Lastly, if you've got specific things that you'd like to see or questions for DARPA or for any of the teams, ping me on Twitter @BotJunkie and I'll happily see what I can do.
  • We Need Software Updates Forever
    Sep 21, 2021 12:00 PM PDT
    Stuart Bradford I recently did some Marie Kondo–inspired housecleaning: Anything that didn't bring me joy got binned. In the process, I unearthed some old gadgets that made me smile. One was my venerable Nokia N95, a proto-smartphone, the first to sport GPS. Another was a craptastic Android tablet—a relic of an era when each year I would purchase the best tablet I could for less than $100 (Australian!), just to see how much you could get for that little. And there was my beloved Sony PlayStation Portable. While I rarely used it, I loved what the PSP represented: a high-powered handheld device, another forerunner of today's smartphone, though one designed for gaming rather than talking. These nifty antiques shared a common problem: Although each booted up successfully, none of them really work anymore. In 2014, Nokia sold off its smartphone division to Microsoft in a fire sale; then Microsoft spiked the whole effort. These moves make my N95 an orphan product from a defunct division of a massive company. Without new firmware, it's essentially useless. My craptastic tablet and PSP similarly need a software refresh. Yet neither of them can log into or even locate the appropriate update servers. You might think that a 15-year-old gaming console wouldn't even be operating, but Sony's build quality is such that, with the exception of a very tired lithium-Ion battery, the unit is in perfect condition. It runs but can't connect to modern Wi-Fi without an update, which it can't access without an update to its firmware (a classic catch-22). I've wasted a few hours trying to work out how to get new firmware on it (and on the tablet), without success. Two perfectly good pieces of electronic gear have become useless, simply for want of software updates. Device makers are apt to drop support for old gadgets faster than the gadgets themselves wear out. Consumers have relied on the good graces of device makers to keep our gadget firmware and software secure and up-to-date. Doing so costs the manufacturer some of its profits. As a result, many of them are apt to drop support for old gadgets faster than the gadgets themselves wear out. This corporate stinginess consigns far too many of our devices to the trash heap before they have exhausted their usability. That's bad for consumers and bad for the planet. It needs to stop. We have seen a global right-to-repair movement emerge from maker communities and start to influence public policy around such things as the availability of spare parts. I'd argue that there should be a parallel right-to-maintain movement. We should mandate that device manufacturers set aside a portion of the purchase price of a gadget to support ongoing software maintenance, forcing them to budget for a future they'd rather ignore. Or maybe they aren't ignoring the future so much as trying to manage it by speeding up product obsolescence, because it typically sparks another purchase. Does this mean Sony and others should still be supporting products nearly two decades old, like my PSP? If that keeps them out of the landfill, I'd say yes: The benefits easily outweigh the costs. The devilish details come in decisions about who should bear those costs. But even if they fell wholly on the purchaser, consumers would, I suspect, be willing to pay a few dollars more for a gadget if that meant reliable access to software for it—indefinitely. Yes, we all want shiny new toys—and we'll have plenty of them—but we shouldn't build that future atop the prematurely discarded remains of our electronic past. This article appears in the October 2021 print issue as "Bricked by Age."
  • How Health Care Organizations Can Thwart Cyberattacks
    Sep 21, 2021 11:00 AM PDT
    Ransomware and other types of cyberattacks are striking health care systems at an increasing rate. More than one in three health care organizations around the world reported ransomware attacks last year, according to a survey of IT professionals by security company Sophos. About 40 percent of the nearly 330 respondents from the health care sector that weren't attacked last year said they expect to be hit in the future. In the United States, the FBI, the Cybersecurity and Infrastructure Security Agency, and the Department of Health and Human Services were so concerned with the increase in cyberattacks on hospitals and other health care providers that in October they issued a joint advisory warning of the "increased and imminent cybercrime threat." But the health care field isn't helpless against cyber threats. The IEEE Standards Association Healthcare and Life Sciences Practice—which is focused on clinical health, the biopharmaceutical value chain, and wellness—recently released Season 2 of the Re-Think Health podcast. The new season features experts from around the world who discuss measures that can help organizations minimize and even prevent attacks. The experts emphasize that cybersecurity is more than an IT concern; they say it needs to be managed from a holistic perspective, aligning employees, technology, and processes within an organization. The six episodes in Cybersecurity for Connected Healthcare Systems: A Global Perspective are as follows: Threat Modeling and Frameworks for Cybersecurity in Connected Health Care Ecosystems. This episode features Florence Hudson, executive director of the Northeast Big Data Innovation Hub. She provides an overview of several programs and initiatives by the IEEE SA Healthcare and Life Sciences Practice. Cracking the Cybersecurity Code to Accelerate Innovation: A View From Australia. Ashish Mahajan, nonexecutive director of the not-for-profit advocacy and research initiative IoTSec Australia, provides insights. He explores vulnerabilities of the data value chain in the Internet of Things ecosystem that could impede innovation in public health, wellness, and health care. Mahajan also chairs the IEEE SA IoT Ecosystem Security Industry Connections program, which aims to work with regulators to promote secure practices. Securing Greater Public Trust in Health Through Risk Mitigation: A North America Perspective. T.R. Kane, a cybersecurity, privacy, and forensics partner at PwC, explains how to strategize and how to respond to vulnerabilities. He offers strategies for managing organizational and patient risk. Uncovering the Great Risk in Security and Privacy of Health Data in Latin America and Beyond. This eye-opening conversation with cybersecurity forensic technologist Andrés Velázquez highlights common global challenges and inherent obstacles. Velázquez is founder and president of Mattica, based in Mexico City. Response and Prevention Strategy in Connected Health: A Perspective From Latin America. Roque Juarez, security intelligence specialist at IBM Mexico, explains how basic principles can be critical to cyber threat management in connected health care systems regardless of whether they are in an emerging or established economy. Roque shares how the COVID-19 pandemic increased the appeal for hackers to breach labs, health care systems, and just about any repository of patient health data and research. Cybersecurity, Trust, and Privacy in Connected Mental Health: A Perspective From Europe. The pandemic has increased the application of digital therapeutics such as mobile apps, games, and virtual reality programs for mental health conditions, according to a guidance document issued in April 2020 by the U.S. Food and Drug Administration. This episode explains opportunities and growing challenges in managing duty of care, security, and privacy with a vulnerable population of patients. MORE EPISODES Season 1 of the podcast is still available. Pain Points of Integrating New Technologies Into an Existing Healthcare Ecosystem features technologists, researchers, and ethicists discussing insights into opportunities and challenges.
  • 7 Revealing Ways AIs Fail
    Sep 21, 2021 08:03 AM PDT
    Artificial intelligence could perform more quickly, accurately, reliably, and impartially than humans on a wide range of problems, from detecting cancer to deciding who receives an interview for a job. But AIs have also suffered numerous, sometimes deadly, failures. And the increasing ubiquity of AI means that failures can affect not just individuals but millions of people. Increasingly, the AI community is cataloging these failures with an eye toward monitoring the risks they may pose. "There tends to be very little information for users to understand how these systems work and what it means to them," says Charlie Pownall, founder of the AI, Algorithmic and Automation Incident & Controversy Repository. "I think this directly impacts trust and confidence in these systems. There are lots of possible reasons why organizations are reluctant to get into the nitty-gritty of what exactly happened in an AI incident or controversy, not the least being potential legal exposure, but if looked at through the lens of trustworthiness, it's in their best interest to do so." This article is part of our special report on AI, “The Great AI Reckoning.” Part of the problem is that the neural network technology that drives many AI systems can break down in ways that remain a mystery to researchers. "It's unpredictable which problems artificial intelligence will be good at, because we don't understand intelligence itself very well," says computer scientist Dan Hendrycks at the University of California, Berkeley. Here are seven examples of AI failures and what current weaknesses they reveal about artificial intelligence. Scientists discuss possible ways to deal with some of these problems; others currently defy explanation or may, philosophically speaking, lack any conclusive solution altogether. 1) Brittleness Chris Philpot Take a picture of a school bus. Flip it so it lays on its side, as it might be found in the case of an accident in the real world. A 2018 study found that state-of-the-art AIs that would normally correctly identify the school bus right-side-up failed to do so on average 97 percent of the time when it was rotated. "They will say the school bus is a snowplow with very high confidence," says computer scientist Anh Nguyen at Auburn University, in Alabama. The AIs are not capable of a task of mental rotation "that even my 3-year-old son could do," he says. Such a failure is an example of brittleness. An AI often "can only recognize a pattern it has seen before," Nguyen says. "If you show it a new pattern, it is easily fooled." There are numerous troubling cases of AI brittleness. Fastening stickers on a stop sign can make an AI misread it. Changing a single pixel on an image can make an AI think a horse is a frog. Neural networks can be 99.99 percent confident that multicolor static is a picture of a lion. Medical images can get modified in a way imperceptible to the human eye so medical scans misdiagnose cancer 100 percent of the time. And so on. One possible way to make AIs more robust against such failures is to expose them to as many confounding "adversarial" examples as possible, Hendrycks says. However, they may still fail against rare " black swan" events. "Black-swan problems such as COVID or the recession are hard for even humans to address—they may not be problems just specific to machine learning," he notes. 2) Embedded Bias Chris Philpot Increasingly, AI is used to help support major decisions, such as who receives a loan, the length of a jail sentence, and who gets health care first. The hope is that AIs can make decisions more impartially than people often have, but much research has found that biases embedded in the data on which these AIs are trained can result in automated discrimination en masse, posing immense risks to society. For example, in 2019, scientists found a nationally deployed health care algorithm in the United States was racially biased, affecting millions of Americans. The AI was designed to identify which patients would benefit most from intensive-care programs, but it routinely enrolled healthier white patients into such programs ahead of black patients who were sicker. Physician and researcher Ziad Obermeyer at the University of California, Berkeley, and his colleagues found the algorithm mistakenly assumed that people with high health care costs were also the sickest patients and most in need of care. However, due to systemic racism, "black patients are less likely to get health care when they need it, so are less likely to generate costs," he explains. After working with the software's developer, Obermeyer and his colleagues helped design a new algorithm that analyzed other variables and displayed 84 percent less bias. "It's a lot more work, but accounting for bias is not at all impossible," he says. They recently drafted a playbook that outlines a few basic steps that governments, businesses, and other groups can implement to detect and prevent bias in existing and future software they use. These include identifying all the algorithms they employ, understanding this software's ideal target and its performance toward that goal, retraining the AI if needed, and creating a high-level oversight body. 3) Catastrophic Forgetting Chris Philpot Deepfakes—highly realistic artificially generated fake images and videos, often of celebrities, politicians, and other public figures—are becoming increasingly common on the Internet and social media, and could wreak plenty of havoc by fraudulently depicting people saying or doing things that never really happened. To develop an AI that could detect deepfakes, computer scientist Shahroz Tariq and his colleagues at Sungkyunkwan University, in South Korea, created a website where people could upload images to check their authenticity. In the beginning, the researchers trained their neural network to spot one kind of deepfake. However, after a few months, many new types of deepfake emerged, and when they trained their AI to identify these new varieties of deepfake, it quickly forgot how to detect the old ones. This was an example of catastrophic forgetting—the tendency of an AI to entirely and abruptly forget information it previously knew after learning new information, essentially overwriting past knowledge with new knowledge. "Artificial neural networks have a terrible memory," Tariq says. AI researchers are pursuing a variety of strategies to prevent catastrophic forgetting so that neural networks can, as humans seem to do, continuously learn effortlessly. A simple technique is to create a specialized neural network for each new task one wants performed—say, distinguishing cats from dogs or apples from oranges—"but this is obviously not scalable, as the number of networks increases linearly with the number of tasks," says machine-learning researcher Sam Kessler at the University of Oxford, in England. One alternative Tariq and his colleagues explored as they trained their AI to spot new kinds of deepfakes was to supply it with a small amount of data on how it identified older types so it would not forget how to detect them. Essentially, this is like reviewing a summary of a textbook chapter before an exam, Tariq says. However, AIs may not always have access to past knowledge—for instance, when dealing with private information such as medical records. Tariq and his colleagues were trying to prevent an AI from relying on data from prior tasks. They had it train itself how to spot new deepfake types while also learning from another AI that was previously trained how to recognize older deepfake varieties. They found this "knowledge distillation" strategy was roughly 87 percent accurate at detecting the kind of low-quality deepfakes typically shared on social media. 4) Explainability Chris Philpot Why does an AI suspect a person might be a criminal or have cancer? The explanation for this and other high-stakes predictions can have many legal, medical, and other consequences. The way in which AIs reach conclusions has long been considered a mysterious black box, leading to many attempts to devise ways to explain AIs' inner workings. "However, my recent work suggests the field of explainability is getting somewhat stuck," says Auburn's Nguyen. Nguyen and his colleagues investigated seven different techniques that researchers have developed to attribute explanations for AI decisions—for instance, what makes an image of a matchstick a matchstick? Is it the flame or the wooden stick? They discovered that many of these methods "are quite unstable," Nguyen says. "They can give you different explanations every time." In addition, while one attribution method might work on one set of neural networks, "it might fail completely on another set," Nguyen adds. The future of explainability may involve building databases of correct explanations, Nguyen says. Attribution methods can then go to such knowledge bases "and search for facts that might explain decisions," he says. 5) Quantifying Uncertainty Chris Philpot In 2016, a Tesla Model S car on autopilot collided with a truck that was turning left in front of it in northern Florida, killing its driver— the automated driving system's first reported fatality. According to Tesla's official blog, neither the autopilot system nor the driver "noticed the white side of the tractor trailer against a brightly lit sky, so the brake was not applied." One potential way Tesla, Uber, and other companies may avoid such disasters is for their cars to do a better job at calculating and dealing with uncertainty. Currently AIs "can be very certain even though they're very wrong," Oxford's Kessler says that if an algorithm makes a decision, "we should have a robust idea of how confident it is in that decision, especially for a medical diagnosis or a self-driving car, and if it's very uncertain, then a human can intervene and give [their] own verdict or assessment of the situation." For example, computer scientist Moloud Abdar at Deakin University in Australia and his colleagues applied several different uncertainty quantification techniques as an AI classified skin-cancer images as malignant or benign, or melanoma or not. The researcher found these methods helped prevent the AI from making overconfident diagnoses. Autonomous vehicles remain challenging for uncertainty quantification, as current uncertainty-quantification techniques are often relatively time consuming, "and cars cannot wait for them," Abdar says. "We need to have much faster approaches." 6) Common Sense Chris Philpot AIs lack common sense—the ability to reach acceptable, logical conclusions based on a vast context of everyday knowledge that people usually take for granted, says computer scientist Xiang Ren at the University of Southern California. "If you don't pay very much attention to what these models are actually learning, they can learn shortcuts that lead them to misbehave," he says. For instance, scientists may train AIs to detect hate speech on data where such speech is unusually high, such as white supremacist forums. However, when this software is exposed to the real world, it can fail to recognize that black and gay people may respectively use the words "black" and "gay" more often than other groups. "Even if a post is quoting a news article mentioning Jewish or black or gay people without any particular sentiment, it might be misclassified as hate speech," Ren says. In contrast, "humans reading through a whole sentence can recognize when an adjective is used in a hateful context." Previous research suggested that state-of-the-art AIs could draw logical inferences about the world with up to roughly 90 percent accuracy, suggesting they were making progress at achieving common sense. However, when Ren and his colleagues tested these models, they found even the best AI could generate logically coherent sentences with slightly less than 32 percent accuracy. When it comes to developing common sense, "one thing we care a lot [about] these days in the AI community is employing more comprehensive checklists to look at the behavior of models on multiple dimensions," he says. 7) Math Chris Philpot Although conventional computers are good at crunching numbers, AIs "are surprisingly not good at mathematics at all," Berkeley's Hendrycks says. "You might have the latest and greatest models that take hundreds of GPUs to train, and they're still just not as reliable as a pocket calculator." For example, Hendrycks and his colleagues trained an AI on hundreds of thousands of math problems with step-by-step solutions. However, when tested on 12,500 problems from high school math competitions, "it only got something like 5 percent accuracy," he says. In comparison, a three-time International Mathematical Olympiad gold medalist attained 90 percent success on such problems "without a calculator," he adds. Neural networks nowadays can learn to solve nearly every kind of problem "if you just give it enough data and enough resources, but not math," Hendrycks says. Many problems in science require a lot of math, so this current weakness of AI can limit its application in scientific research, he notes. It remains uncertain why AI is currently bad at math. One possibility is that neural networks attack problems in a highly parallel manner like human brains, whereas math problems typically require a long series of steps to solve, so maybe the way AIs process data is not as suitable for such tasks, "in the same way that humans generally can't do huge calculations in their head," Hendrycks says. However, AI's poor performance on math "is still a niche topic: There hasn't been much traction on the problem," he adds. Special Report: The Great AI Reckoning READ NEXT: How the U.S. Army Is Turning Robots Into Team Players Or see the full report for more articles on the future of AI.
  • DARPA SubT Finals: Meet the Teams
    Sep 21, 2021 05:52 AM PDT
    This is it! This week, we're at the DARPA SubTerranean Challenge Finals in Louisville KY, where more than two dozen Systems Track and Virtual Track teams will compete for millions of dollars in prize money and being able to say "we won a DARPA challenge," which is of course priceless. We've been following SubT for years, from Tunnel Circuit to Urban Circuit to Cave (non-) Circuit. For a recent recap, have a look at this post-cave pre-final article that includes an interview with SubT Program Manager Tim Chung, but if you don't have time for that, the TLDR is that this week we're looking at both a Virtual Track as well as a Systems Track with physical robots on a real course. The Systems Track teams spent Monday checking in at the Louisville Mega Cavern competition site, and we asked each team to tell us about how they've been preparing, what they think will be most challenging, and what makes them unique. Team CERBERUS Team CERBERUS CERBERUS Country USA, Switzerland, United Kingdom, Norway Members University of Nevada, Reno ETH Zurich, Switzerland University of California, Berkeley Sierra Nevada Corporation Flyability, Switzerland Oxford Robotics Institute, United Kingdom Norwegian University for Science and Technology (NTNU), Norway Robots TBA Follow Team Website @CerberusSubt Q&A: Team Lead Kostas Alexis How have you been preparing for the SubT Final? First of all this year's preparation was strongly influenced by Covid-19 as our team spans multiple countries, namely the US, Switzerland, Norway, and the UK. Despite the challenges, we leveled up both our weekly shake-out events and ran a 2-month team-wide integration and testing activity in Switzerland during July and August with multiple tests in diverse underground settings including multiple mines. Note that we bring a brand new set of 4 ANYmal C robots and a new generation of collision-tolerant flying robots so during this period we further built new hardware. What do you think the biggest challenge of the SubT Final will be? We are excited to see how the combination of vastly large spaces available in Mega Caverns can be combined with very narrow cross-sections as DARPA promises and vertical structures. We think that terrain with steep slopes and other obstacles, complex 3D geometries, as well as the dynamic obstacles will be the core challenges. What is one way in which your team is unique, and why will that be an advantage during the competition? Our team coined early on the idea of legged and flying robot combination. We have remained focused on this core vision of ours and also bring fully own-developed hardware for both legged and flying systems. This is both our advantage and - in a way - our limitation as we spend a lot of time in its development. We are fully excited about the potential we see developing and we are optimistic that this will be demonstrated in the Final Event! Team Coordinated Robotics Team Coordinated Robotics Coordinated Robotics Country USA Members California State University Channel Islands Oke Onwuka Sequoia Middle School Robots TBA Q&A: Team Lead Kevin Knoedler How have you been preparing for the SubT Final? Coordinated Robotics has been preparing for the SubT Final with lots of testing on our team of robots. We have been running them inside, outside, day, night and all of the circumstances that we can come up with. In Kentucky we have been busy updating all of the robots to the same standard and repairing bits of shipping damage before the Subt Final. What do you think the biggest challenge of the SubT Final will be? The biggest challenge for us will be pulling all of the robots together to work as a team and make sure that everything is communicating together. We did not have lab access until late July and so we had robots at individuals homes, but were generally only testing one robot at a time. What is one way in which your team is unique, and why will that be an advantage during the competition? Coordinated Robotics is unique in a couple of different ways. We are one of only two unfunded teams so we take a lower budget approach to solving lots of the issues and that helps us to have some creative solutions. We are also unique in that we will be bringing a lot of robots (23) so that problems with individual robots can be tolerated as the team of robots continues to search. Team CoSTAR Team CoSTAR CoSTAR Country USA, South Korea, Sweden Members Jet Propulsion Laboratory California Institute of Technology Massachusetts Institute of Technology KAIST, South Korea Lulea University of Technology, Sweden Robots TBA Follow Team Website Q&A: Caltech Team Lead Joel Burdick How have you been preparing for the SubT Final? Since May, the team has made 4 trips to a limestone cave near Lexington Kentucky (and they are just finishing a week-long "game" there yesterday). Since February, parts or all of the team have been testing 2-3 days a week in a section of the abandoned Subway system in downtown Los Angeles. What do you think the biggest challenge of the SubT Final will be? That will be a tough one to answer in advance. The expected CoSTAR-specific challenges are of course the complexity of the test-site that DARPA has prepared, fatigue of the team, and the usual last-minute hardware failures: we had to have an entire new set of batteries for all of our communication nodes FedExed to us yesterday. More generally, we expect the other teams to be well prepared. Speaking only for myself, I think there will be 4-5 teams that could easily win this competition. What is one way in which your team is unique, and why will that be an advantage during the competition? Previously, our team was unique with our Boston Dynamic legged mobility. We've heard that other teams maybe using Spot quadrupeds as well. So, that may no longer be a uniqueness. We shall see! More importantly, we believe our team is unique in the breadth of the participants (university team members from U.S., Europe, and Asia). Kind of like the old British empire: the sun never sets on the geographic expanse of Team CoSTAR. Team CSIRO Data61 Team CSIRO Data61 CSIRO Data61 Country Australia, USA Members Commonwealth Scientific and Industrial Research Organisation, Australia Emesent, Australia Georgia Institute of Technology Robots TBA Follow Team Website Twitter Q&A: SubT Principal Investigator Navinda Kottege How have you been preparing for the SubT Final? Test, test, test. We've been testing as often as we can, simulating the competition conditions as best we can. We're very fortunate to have an extensive site here at our CSIRO lab in Brisbane that has enabled us to construct quite varied tests for our full fleet of robots. We have also done a number of offsite tests as well. After going through the initial phases, we have converged on a good combination of platforms for our fleet. Our work horse platform from the Tunnel circuit has been the BIA5 ATR tracked robot. We have recently added Boston Dynamics Spot quadrupeds to our fleet and we are quite happy with their performance and the level of integration with our perception and navigation stack. We also have custom designed Subterra Navi drones from Emesent. Our fleet consists of two of each of these three platform types. We have also designed and built a new 'Smart node' for communication with the Rajant nodes. These are dropped from the tracked robots and automatically deploy after a delay by extending out ground plates and antennae. As described above, we have been doing extensive integration testing with the full system to shake out bugs and make improvements. What do you think the biggest challenge of the SubT Final will be? The biggest challenge is the unknown. It is always a learning process to discover how the robots respond to new classes of obstacle; responding to this on the fly in a new environment is extremely challenging. Given the format of two preliminary runs and one prize run, there is little to no margin for error compared to previous circuit events where there were multiple runs that contributed to the final score. Any significant damage to robots during the preliminary runs would be difficult to recover from to perform in the final run. What is one way in which your team is unique, and why will that be an advantage during the competition? Our fleet uses a common sensing, mapping and navigation system across all robots, built around our Wildcat SLAM technology. This is what enables coordination between robots, and provides the accuracy required to locate detected objects. This had allowed us to easily integrate different robot platforms into our fleet. We believe this 'homogenous sensing on heterogenous platforms' paradigm gives us a unique advantage in reducing overall complexity of the development effort for the fleet and also allowing us to scale our fleet as needed. Having excellent partners in Emesent and Georgia Tech and having their full commitment and support is also a strong advantage for us. Team CTU-CRAS-NORLAB Team CTU-CRAS-NORLAB CTU-CRAS-NORLAB Country Czech Republic, Canada Members Czech Technological University, Czech Republic Université Laval, Canada Robots TBA Follow Team Website Twitter Q&A: Team Lead Tomas Svoboda How have you been preparing for the SubT Final? We spent most of the time preparing new platforms as we made a significant technology update. We tested the locomotion and autonomy of the new platforms in Bull Rock Cave, one of the largest caves in Czechia. We also deployed the robots in an old underground fortress to examine the system in an urban-like underground environment. The very last weeks were, however, dedicated to integration tests and system tuning. What do you think the biggest challenge of the SubT Final will be? Hard to say, but regarding the expected environment, the vertical shafts might be the most challenging since they are not easy to access to test and tune the system experimentally. They would also add challenges to communication. What is one way in which your team is unique, and why will that be an advantage during the competition? Not sure about the other teams, but we plan to deploy all kinds of ground vehicles, tracked, wheeled, and legged platforms accompanied by several drones. We hope the diversity of the platform types would be beneficial for adapting to the possible diversity of terrains and underground challenges. Besides, we also hope the tuned communication would provide access to robots in a wider range than the last time. Optimistically, we might keep all robots connected to the communication infrastructure built during the mission, albeit the bandwidth is very limited, but should be sufficient for artifacts reporting and high-level switching of the robots' goals and autonomous behavior. Team Explorer Team Explorer Explorer Country USA Members Carnegie Mellon University Oregon State University Robots TBA Follow Team Website Facebook Q&A: Team Co-Lead Sebastian Scherer How have you been preparing for the SubT Final? Since we expect DARPA to have some surprises on the course for us, we have been practicing in a wide range of different courses around Pittsburgh including an abandoned hospital complex, a cave and limestone and coal mines. As the finals approached, we were practicing at these locations nearly daily, with debrief and debugging sessions afterward. This has helped us find the advantages of each of the platforms, ways of controlling them, and the different sensor modalities. What do you think the biggest challenge of the SubT Final will be? For our team the biggest challenges are steep slopes for the ground robots and thin loose obstacles that can get sucked into the props for the drones as well as narrow passages. What is one way in which your team is unique, and why will that be an advantage during the competition? We have developed a heterogeneous team for SubT exploration. This gives us an advantage since there is not a single platform that is optimal for all SubT environments. Tunnels are optimal for roving robots, urban environments for walking robots, and caves for flying. Our ground robots and drones are custom-designed for navigation in rough terrain and tight spaces. This gives us an advantage since we can get to places not reachable by off-the-shelf platforms. Team MARBLE Team MARBLE MARBLE Country USA Members University of Colorado, Boulder University of Colorado, Denver Scientific Systems Company, Inc. University of California, Santa Cruz Robots TBA Follow Team Twitter Q&A: Project Engineer Gene Rush How have you been preparing for the SubT Final? Our team has worked tirelessly over the past several months as we prepare for the SubT Final. We have invested most of our time and energy in real-world field deployments, which help us in two major ways. First, it allows us to repeatedly test the performance of our full autonomy stack, and second, it provides us the opportunity to emphasize Pit Crew and Human Supervisor training. Our PI, Sean Humbert, has always said "practice, practice, practice." In the month leading up to the event, we stayed true to this advice by holding 10 deployments across a variety of environments, including parking garages, campus buildings at the University of Colorado Boulder, and the Edgar Experimental Mine. What do you think the biggest challenge of the SubT Final will be? I expect the most difficult challenge will is centered around autonomous high-level decision making. Of course, mobility challenges, including treacherous terrain, stairs, and drop offs will certainly test the physical capabilities of our mobile robots. However, the scale of the environment is so great, and time so limited, that rapidly identifying the areas that likely have human survivors is vitally important and a very difficult open challenge. I expect most teams, ours included, will utilize the intuition of the Human Supervisor to make these decisions. What is one way in which your team is unique, and why will that be an advantage during the competition? Our team has pushed on advancing hands-off autonomy, so our robotic fleet can operate independently in the worst case scenario: a communication-denied environment. The lack of wireless communication is relatively prevalent in subterranean search and rescue missions, and therefore we expect DARPA will be stressing this part of the challenge in the SubT Final. Our autonomy solution is designed in such a way that it can operate autonomously both with and without communication back to the Human Supervisor. When we are in communication with our robotic teammates, the Human Supervisor has the ability to provide several high level commands to assist the robots in making better decisions. Team Robotika Team Robotika Robotika Country Czech Republic, USA, Switzerland Members Robotika International, Czech Republic and United States, Czech Republic Czech University of Life Science, Czech Republic Centre for Field Robotics, Czech Republic Cogito Team, Switzerland Robots Two wheeled robots Follow Team Website Twitter Q&A: Team Lead Martin Dlouhy How have you been preparing for the SubT Final? Our team participates in both Systems and Virtual tracks. We were using the virtual environment to develop and test our ideas and techniques and once they were sufficiently validated in the virtual world, we would transfer these results to the Systems track as well. Then, to validate this transfer, we visited a few underground spaces (mostly caves) with our physical robots to see how they perform in the real world. What do you think the biggest challenge of the SubT Final will be? Besides the usual challenges inherent to the underground spaces (mud, moisture, fog, condensation), we also noticed the unusual configuration of the starting point which is a sharp downhill slope. Our solution is designed to be careful about going on too steep slopes so our concern is that as things stand, the robots may hesitate to even get started. We are making some adjustments in the remaining time to account for this. Also, unlike the environment in all the previous rounds, the Mega Cavern features some really large open spaces. Our solution is designed to expect detection of obstacles somewhere in the vicinity of the robot at any given point so the concern is that a large open space may confuse its navigational system. We are looking into handling such a situation better as well. What is one way in which your team is unique, and why will that be an advantage during the competition? It appears that we are unique in bringing only two robots into the Finals. We have brought more into the earlier rounds to test different platforms and ultimately picked the two we are fielding this time as best suited for the expected environment. A potential benefit for us is that supervising only two robots could be easier and perhaps more efficient than managing larger numbers.
  • Solar and Battery Companies Rattle Utility Powerhouses
    Sep 20, 2021 08:15 AM PDT
    All eyes these days may be on Elon Musk's space venture—which has just put people in orbit—but here on Earth you can now get your monthly electric bill courtesy of a different Musk enterprise. Tesla and its partner Octopus Energy Germany recently rolled out retail utility services in two large German states. It's being marketed as the "Tesla Energy Plan," and is available to any individual household in this region of 24 million people that has a solar panel system, a grid connection—and a Tesla powerwall, the Palo Alto firm's gigafactory-made 13.5 kWh battery wall unit. The German initiative comes on the heels of a similar rollout through Octopus Energy last November in the United Kingdom. It's too soon to say if these are the nascent strands of a "giant distributed utility," an expression Musk has long talked up, the meaning of which is not yet clear. Analysts and power insiders sketch scenes including interconnected local renewable grids that draw on short-duration battery storage (including the small batteries in electric vehicles in a garage, models for which Tesla just happens to make) combined with multi-day storage for power generated by wind and solar. For bigger national grids it gets more complicated. Even so, Tesla also now has gear on the market that institutional battery storage developers can use to run load-balancing trade operations: the consumer won't see those, but it's part of ongoing changes as renewables become more important in the power game. Being able to get a Tesla-backed power bill in the mailbox, though—that's grabbing attention. And more broadly speaking, the notion of what is and isn't a utility is in flux. "Over the last five to 10 years we have seen an uptick in new entrants providing retail energy services," says Albert Cheung, head of global analysis at BloombergNEF. "It is now quite common to see these types of companies gain significant market share without necessarily owning any of their own generation or network assets at all." A decade ago it became possible to get your electricity in the UK from a department store chain (though with the actual power supplied first by a Scottish utility and—as of 2018—arranged and managed by Octopus Energy). As Tesla and other makers of home energy storage systems ramp up production for modular large-scale lithium-ion batteries that can be stacked together in industrial storage facilities, new wrinkles are coming to the grid. "There are simply going to be more and different business models out there," Cheung says. "There is going to be value in distributed energy resources at the customer's home; Whether that is a battery, an electric vehicle charger, a heat pump or other forms of flexible load, and managing these in a way that provides value to the grid will create revenue opportunities." Tesla Gigafactory site taking shape in Grünheide, Germany in June 2021. It is due to open in late 2021 or early 2022. Michael Dumiak Tesla the battery-maker, with its giant new production plant nearing completion in Berlin, may be in position to supply a variety of venues with its wall-sized and cargo-container-sized units: As it does so, its controversial bet in first backing and then absorbing panel producer Solar City may start to look a little different. Harmony Energy seems pretty pleased. The UK-based energy developer's just broken ground on a new four-acre battery storage site outside London, its third such site. Its second just came online with 68 MWh storage capacity and a 34 MW peak, with the site comprising 28 Tesla Megapack batteries. Harmony expects to be at over a gigawatt of live, operating output in the next three to four years. The Harmony enterprise works with the UK national grid, however—that's different from Octopus's German and UK retail initiatives. Both Harmony and Octopus depend on trading and energy network management software platforms, and Tesla works with both. But while Octopus has its own in-house management platform—Kraken—Harmony engages Tesla's Autobidder. Peter Kavanagh, Harmony's CEO, says his firm pays Tesla to operate Autobidder on its behalf—Tesla is fully licensed to trade in the UK and is an approved utility there. The batteries get charged when power is cheap; when there's low wind and no sun, energy prices may start to spike, and the batteries can discharge the power back into the grid, balancing the constant change of supply and demand, and trading on the difference to make a business. A load-balancing trading operation is not quite the same as mainlining renewables to light a house. On any national grid, once the energy is in there, it's hard to trace the generating source—some of it will come from fossil fuels. But industrial-scale energy storage is crucial to any renewable operation: the wind dies down, the sun doesn't always shine. "Whether it's batteries or some other energy storage technology, it is key to hitting net zero carbon emissions," Kavanagh says. "Without it, you are not going to get there." Battery research and development is burgeoning far beyond Tesla, and the difficult hunt is on to move past lithium ion. And it's not just startups and young firms in the mix: Established utility giants—the Pacific Gas & Electrics of the world, able to generate as well as retail power—are also adding battery storage, and at scale. In Germany, the large industrial utility RWE started its own battery unit and is now operating small energy storage sites in Germany and in Arizona. Newer entrants, potential energy powerhouses, are on the rise in Italy, Spain and Denmark. The Tesla Energy plan does have German attention though, of media and energy companies alike. It's also of note that Tesla is behind the very large battery at Australia's Hornsdale Power Reserve. One German pundit imagined Octopus's Kraken management platform as a "monstrous octopus with millions of tentacles," linking a myriad of in-house electric storage units to form a huge virtual power plant. That would be something to reckon with.
  • Help Build the Future of Assistive Technology
    Sep 20, 2021 05:09 AM PDT
    This article is sponsored by California State University, Northridge (CSUN). Your smartphone is getting smarter. Your car is driving itself. And your watch tells you when to breathe. That, as strange as it might sound, is the world we live in. Just look around you. Almost every day, there's a better or more convenient version of the latest gadget, device, or software. And that's only on the commercial end. The medical and rehabilitative tech is equally impressive — and arguably far more important. Because for those with disabilities, assistive technologies mean more than convenience. They mean freedom. So, what is an assistive technology (AT), and who designs it? The term might be new to you, but you're undoubtedly aware of many: hearing aids, prosthetics, speech-recognition software (Hey, Siri), even the touch screen you use each day on your cell phone. They're all assistive technologies. AT, in its most basic form, is anything that helps a person achieve enhanced performance, improved function, or accelerated access to information. A car lets you travel faster than walking; a computer lets you process data at an inhuman speed; and a search engine lets you easily find information. CSUN Master of Science in Assistive Technology Engineering The fully online M.S. in Assistive Technology Engineering program can be completed in less than two years and allows you to collaborate with other engineers and AT professionals. GRE is not required and financial aid is available. Request more information about the program here. That's the concept – in a simplified form, of course. The applications, however, are vast and still expanding. In addition to mechanical products and devices, the field is deeply involved in artificial intelligence, machine learning, and neuroscience. Brain machine interfaces, for instance, allow users to control prosthetics with thought alone; and in some emergency rooms, self-service kiosks can take your blood pressure, pulse and weight, all without any human intervention. These technologies, and others like them, will only grow more prevalent with time – as will the need for engineers to design them. Those interested in the field typically enter biomedical engineering programs. These programs, although robust in design, focus often on hardware, teaching students how to apply engineering principles to medicine and health care. What many lack, however, is a focus on the user. But that's changing. Some newer programs, many of them certificates, employ a more user-centric model. One recent example is the Master of Science in Assistive Technology Engineering at California State University, Northridge (CSUN). The degree, designed in collaboration with industry professionals, is a hybrid of sorts, focusing as much on user needs as on the development of new technologies. CSUN, it should be noted, is no newcomer to the field. For more than three decades, the university has hosted the world's largest assistive technology conference. To give you an idea, this year's attendees included Google, Microsoft, Hulu, Amazon, and the Central Intelligence Agency. The university is also home to a sister degree, the Master of Science in Assistive Technology and Human Services, which prepares graduates to assist and train AT users. As you can imagine, companies are aggressively recruiting engineers with this cross-functional knowledge. Good UX design is universally desired, as it's needed for both optimal function and, often, ADA compliance. In addition to mechanical devices, the field of Assistive Technology is deeply involved in artificial intelligence, machine learning, and neuroscience The field has implications in war as well – both during and after. Coming as no surprise, the military is investing heavily in AT hardware and research. Why? On the most basic level, the military is interested in rehabilitating combat veterans. Assistive technologies, such as prosthetic limbs, enable those wounded in combat to pursue satisfying lives in the civilian world. Beyond that, assistive technology is a core part of the military's long-term strategic plan. Wearable electronics, such as VR headsets and night vision goggles, both fit within the military's expanding technological horizon, as do heads-up displays, exoskeletons and drone technologies. The Future of Assistive Technology So, what does the future have in store for AT? We'll likely see more and better commercial technologies designed for entertainment. Think artificial realities with interactive elements in the real world (a whale floating by your actual window, not a simulated one). Kevin Kelly of Wired Magazine refers to this layered reality as the "Mirrorworld." And according to him, it's going to spark the next tech platform. Imagine Facebook in the Matrix... Or, come to think of it, don't. An increasing number of mobile apps, such as those able to detect Parkinson's disease, will also hit the market. As will new biomedical hardware, like brain and visual implants. Fortunately, commercial innovations often drive medical ones as well. And as we see an uptick in entertainment, we'll see an equal surge in medicine, with new technologies – things we haven't even considered yet – empowering those in need. Help build the future of assistive technology! Visit CSUN's Master of Science in Assistive Technology Engineering site to learn more about the program or request more information here.
  • Video Friday: Preparing for the SubT Final
    Sep 17, 2021 08:23 AM PDT
    Video Friday is your weekly selection of awesome robotics videos, collected by your friends at IEEE Spectrum robotics. We'll also be posting a weekly calendar of upcoming robotics events for the next few months; here's what we have so far (send us your events!): DARPA SubT Finals – September 21-23, 2021 – Louisville, KY, USA WeRobot 2021 – September 23-25, 2021 – [Online Event] IROS 2021 – September 27-1, 2021 – [Online Event] Robo Boston – October 1-2, 2021 – Boston, MA, USA WearRAcon Europe 2021 – October 5-7, 2021 – [Online Event] ROSCon 2021 – October 20-21, 2021 – [Online Event] Silicon Valley Robot Block Party – October 23, 2021 – Oakland, CA, USA Let us know if you have suggestions for next week, and enjoy today's videos. Team Explorer, the SubT Challenge entry from CMU and Oregon State University, is in the last stage of preparation for the competition this month inside the Mega Caverns cave complex in Louisville, Kentucky. [ Explorer ] Team CERBERUS is looking good for the SubT Final next week, too. Autonomous subterranean exploration with the ANYmal C Robot inside the Hagerbach underground mine [ ARL ] I'm still as skeptical as I ever was about a big and almost certainly expensive two-armed robot that can do whatever you can program it to do (have fun with that) and seems to rely on an app store for functionality. [ Unlimited Robotics ] Project Mineral is using breakthroughs in artificial intelligence, sensors, and robotics to find ways to grow more food, more sustainably. [ Mineral ] Not having a torso or anything presumably makes this easier. Next up, Digit limbo! [ Hybrid Robotics ] Paric completed layout of a 500 unit apartment complex utilizing the Dusty FieldPrinter solution. Autonomous layout on the plywood deck saved weeks worth of schedule, allowing the panelized walls to be placed sooner. [ Dusty Robotics ] Spot performs inspection in the Kidd Creek Mine, enabling operators to keep their distance from hazards. [ Boston Dynamics ] Digit's engineered to be a multipurpose machine. Meaning, it needs to be able to perform a collection of tasks in practically any environment. We do this by first ensuring the robot's physically capable. Then we help the robot perceive its surroundings, understand its surroundings, then reason a best course of action to navigate its environment and accomplish its task. This is where software comes into play. This is early AI in action. [ Agility Robotics ] This work proposes a compact robotic limb, AugLimb, that can augment our body functions and support the daily activities. The proposed device can be mounted on the user's upper arm, and transform into compact state without obstruction to wearers. [ AugLimb ] Ahold Delhaize and AIRLab need the help of academics who have knowledge of human-robot interactions, mobility, manipulation, programming, and sensors to accelerate the introduction of robotics in retail. In the AIRLab Stacking challenge, teams will work on algorithms that focus on smart retail applications, for example, automated product stacking. [ PAL Robotics ] Leica, not at all well known for making robots, is getting into the robotic reality capture business with a payload for Spot and a new drone. Introducing BLK2FLY: Autonomous Flying Laser Scanner [ Leica BLK ] As much as I like Soft Robotics, I'm maybe not quite as optimistic as they are about the potential for robots to take over quite this much from humans in the near term. [ Soft Robotics ] Over the course of this video, the robot gets longer and longer and longer. [ Transcend Robotics ] This is a good challenge: attach a spool of electrical tape to your drone, which can unpredictably unspool itself and make sure it doesn't totally screw you up. [ UZH ] Two interesting short seminars from NCCR Robotics, including one on autonomous racing drones and "neophobic" mobile robots. Dario Mantegazza: Neophobic Mobile Robots Avoid Potential Hazards [ NCCR ] This panel on Synergies between Automation and Robotics comes from ICRA 2021, and once you see the participant list, I bet you'll agree that it's worth a watch. [ ICRA 2021 ] CMU RI Seminars are back! This week we hear from Andrew E. Johnson, a Principal Robotics Systems Engineer in the Guidance and Control Section of the NASA Jet Propulsion Laboratory, on "The Search for Ancient Life on Mars Began with a Safe Landing." Prior mars rover missions have all landed in flat and smooth regions, but for the Mars 2020 mission, which is seeking signs of ancient life, this was no longer acceptable. Terrain relief that is ideal for the science obviously poses significant risks for landing, so a new landing capability called Terrain Relative Navigation (TRN) was added to the mission. This talk will describe the scientific goals of the mission, the Terrain Relative Navigation system design and the successful results from landing on February 18th, 2021. [ CMU RI Seminar ]
  • China’s Mars Helicopter to Support Future Rover Exploration
    Sep 17, 2021 08:17 AM PDT
    The first ever powered flight by an aircraft on another planetary took place in April when NASA's Ingenuity helicopter, delivered to the Red Planet along with Perseverance rover, but the idea has already taken off elsewhere. Earlier this month a prototype "Mars surface cruise drone system" developed by a team led by Bian Chunjiang at China's National Space Science Center (NSSC) in Beijing gained approval for further development. Like Ingenuity, which was intended purely as a technology demonstration, it uses two sets of blades on a single rotor mast to provide lift for vertical take-offs and landings in the very thin Martian atmosphere, which is around 1% the density of Earth's. The team did consider a fixed wing approach, which other space-related research institutes in China have been developing, but found the constraints related to size, mass, power and lift best met by the single rotor mast approach. Solar panels charge Ingenuity's batteries enough to allow one 90-second flight per Martian day. The NSSC team is however considering adopting wireless charging through the rover, or a combination of both power systems. The total mass is 2.1 kilograms, slightly heavier than the 1.8-kg Ingenuity. It would fly at an altitude of 5-10 meters, reaching speeds of around 300 meters per minute, with a possible duration of 3 minutes per flight. Limitations include energy consumption and temperature control. According to an article published by China Science Daily, Bian proposed development of a helicopter to help guide a rover in March 2019, which was then accepted in June that year. The idea is that by imaging areas ahead the rover could then better select routes which avoid the otherwise unseen areas that restrict and pose challenges to driving. The small craft's miniature multispectral imaging system may also detect scientifically valuable targets, such as evidence of notable compounds, that would otherwise be missed, deliver preliminary data and direct the rover for more detailed observations. The next steps, Bian said, will be developing the craft so as to be able to operate in the very low atmospheric pressure and frigid temperatures of Mars as well as the dust environment and other complex environmental variables. Bian also notes that to properly support science and exploration goals the helicopter design life must be at least a few months or even beyond a year on Mars. To properly test the vehicle, these conditions will have to be simulated here on Earth. Bian says China does not currently have facilities that can meet all of the parameters. Faced with similar challenges for Ingenuity, Caltech graduate students built a custom wind tunnel for testing, and the NSSC team may likewise need to take a bespoke approach. "The next 5 to 6 years are a window for research." Bian said. "We hope to overcome these technical problems and allow the next Mars exploration mission to carry a drone on Mars." When the Mars aircraft could be deployed on Mars is unknown. China's first Mars rover landed in May, but there is no backup vehicle, unlike its predecessor lunar rover missions. The country's next interplanetary mission is expected to be a complex and unprecedented Mars sample-return launching around 2028-2030. Ingenuity's first flight was declared by NASA to be a "Wright Brothers moment." Six years after the 1903 Wright Flyer, Chinese-born Feng Ru successfully flew his own biplane. Likewise, in the coming years, China will be looking to carry out its own powered flight on another planet.
  • New Fuel Cell Tech Points Toward Zero-Emission Trains
    Sep 17, 2021 06:00 AM PDT
    This article is part of our exclusive IEEE Journal Watch series in partnership with IEEE Xplore. Diesel and steam-powered trains have been transporting passengers and cargo around the world for more than 200 years—all the while releasing greenhouse gas emissions into the atmosphere. In the hopes of a greener future, many countries and companies are eyeing more renewable sources of locomotion. The Pittsburgh-based company Wabtec recently unveiled a battery-electric hybrid train that they say can reduce emissions "by double digits per train." More ambitiously, some are developing hydrogen-powered trains, which rather than emitting greenhouse gases, only produce water vapor and droplets. The technology has the potential to help countries meet greenhouse gas reduction targets and slow the progression of climate change. But, producing electricity from hydrogen comes with its own challenges. For example, the fuel cells require additional heavy converters to manage their wide voltage range. The weight of these bulky converters ultimately reduces the range of the train. In a recent advancement, researchers in the UK have designed a new converter that is substantially lighter and more compact than state-of-the art hydrogen cell converters. They describe the new design in study published August 25 in IEEE Transactions on Industrial Electronics. Pietro Tricoli, a professor at the University of Birmingham, was involved in the study. He notes that lighter converters are needed to help maximize the range that hydrogen powered trains can travel. Therefore his team developed the newer, lighter converter, which they describe in their paper as "ground-breaking." It uses semiconductor devices to draw energy in a controlled way from the fuel cells and deliver it to the train's motors. "Our converter directly manages any voltage variations in the fuel cells, without affecting the motor currents. A conventional system would require two separate converters to achieve this," explains Tricoli. With the power converted to AC, the motors of a train can benefit from regenerative braking, whereby energy is harvested and recycled when the train is decelerating. The researchers first tested their design through simulations, and then developed validated it through a small-scale laboratory prototype representing the traction system of a train. The results confirm that the new converter can facilitate desirable speeds and accelerations, as well as achieve regenerative braking. Left: A prototype of the new hydrogen cell converter. Right: A module used at the heart of the converter.Ivan Krastev "The main strength of the converter is the reduction of volume and weight comparted to the state of the art [converters for hydrogen fuel cells]," explains Tricoli. The main drawback, he says, is that the new converter design requires more semiconductor devices, as well as more complex circuitry and monitoring systems. Tricoli says there's still plenty of work ahead optimizing the system, ultimately, toward a full-scale prototype. "The current plan is to engage with train manufacturers and manufacturers of traction equipment to build a second [prototype] for a hydrogen train," he says. This past spring marked an exciting milestone when, upon the completion of a 538-day trial period, two hydrogen-powered trains successfully transported passengers across 180,000 kilometers in Germany—while emitting zero vehicle emissions. As more advancements in hydrogen technology such as the above are made, more increasingly efficient hydrogen-powered trains become possible. All aboard!
  • Q&A With Co-Creator of the 6502 Processor
    Sep 16, 2021 11:00 AM PDT
    Few people have seen their handiwork influence the world more than Bill Mensch. He helped create the legendary 8-bit 6502 microprocessor, launched in 1975, which was the heart of groundbreaking systems including the Atari 2600, Apple II, and Commodore 64. Mensch also created the VIA 65C22 input/output chip—noted for its rich features and which was crucial to the 6502's overall popularity—and the second-generation 65C816, a 16-bit processor that powered machines such as the Apple IIGS, and the Super Nintendo console. Many of the 65x series of chips are still in production. The processors and their variants are used as microcontrollers in commercial products, and they remain popular among hobbyists who build home-brewed computers. The surge of interest in retrocomputing has led to folks once again swapping tips on how to write polished games using the 6502 assembly code, with new titles being released for the Atari, BBC Micro, and other machines. Mensch, an IEEE senior life member, splits his time between Arizona and Colorado, but folks in the Northeast of the United States will have the opportunity to see him as a keynote speaker at the Vintage Computer Festival in Wall, N.J., on the weekend of 8 October. In advance of Mensch's appearance, The Institute caught up with him via Zoom to talk about his career. This interview had been condensed and edited for clarity. The Institute: What drew you into engineering? Bill Mensch: I went to Temple University [in Philadelphia] on the recommendation of a guidance counselor. When I got there I found they only had an associate degree in engineering technology. But I didn't know what I was doing, so I thought: Let's finish up that associate degree. Then I got a job [in 1967] as a technician at [Pennsylvania TV maker] Philco-Ford and noticed that the engineers were making about twice as much money. I also noticed I was helping the engineers figure out what Motorola was doing in high-voltage circuits—which meant that Motorola was the leader and Philco was the follower. So I went to the University of Arizona, close to where Motorola was, got my engineering degree [in 1971] and went to work for Motorola. TI: How did you end up developing the 6502? BM: Chuck Peddle approached me. He arrived at Motorola two years after I started. Now, this has not been written up anywhere that I'm aware of, but I think his intention was to raid Motorola for engineers. He worked with me on the peripheral interface chip (PIA) and got to see me in action. He decided I was a young, egotistical engineer who was just the right kind to go with his ego. So Chuck and I formed a partnership of sorts. He was the system engineer, and I was the semiconductor engineer. We tried to start our own company [with some other Motorola engineers] and when that didn't happen, we joined an existing [semiconductor design] company, called MOS Technology, in Pennsylvania in 1974. That's where we created the 6501 and 6502 [in 1975], and I designed the input/output chips that went with it. The intention was to [develop a US $20 microprocessor to] compete with the Intel 4040 microcontroller chipset, which sold for about $29 at the time. We weren't trying to compete with the 6800 or the 8080 [chips designed for more complex microcomputer systems]. TI: The 6502 did become the basis of a lot of microcomputer systems, and if you look at contemporary programmer books, they often talk about the quirks of the 6502's architecture and instruction set compared with other processors. What drove those design decisions? BM: Rod Orgill and I had completed the designs of a few microprocessors before the 6501/6502. In other words, Rod and I already knew what was successful in an instruction set. And lower cost was key. So we looked at what instructions we really needed. And we figured out how to have addressable registers by using zero page [the first 256 bytes in RAM]. So you can have one byte for the op code and one byte for the address, and [the code is compact and fast]. There are limitations, but compared to other processors, zero page was a big deal. There is a love for this little processor that's undeniable. TI: A lot of pages in those programming books are devoted to explaining how to use the versatile interface adapter (VIA) chip and its two I/O ports, on-board timers, a serial shift register, and so on. Why so many features? BM: I had worked on the earlier PIA chip at Motorola. That meant I understood the needs of real systems in real-world implementations. [While working at MOS] Chuck, Wil Mathis, our applications guy, and I were eating at an Arby's one day, and we talked about doing something beyond the PIA. And they were saying, "We'd like to put a couple of timers on it. We'd like a serial port," and I said, "Okay, we're going to need more register select lines." And our notes are on an Arby's napkin. And I went off and designed it. Then I had to redesign it to make it more compatible with the PIA. I also made a few changes at Apple's request. What's interesting about the VIA is that it's the most popular chip we sell today. I'm finding out more and more about how it was used in different applications. TI: After MOS Technology, in 1978 you founded The Western Design Center, where you created the 65C816 CPU. The creators of the ARM processor credit a visit to WDC as giving them the confidence to design their own chip. Do you remember that visit? BM: Vividly! Sophie Wilson and Steve Furber visited me and talked to me about developing a 32-bit chip. They wanted to leapfrog what Apple was rumored to be up to. But I was just finishing up the '816, and I didn't want to change horses. So when they [had success with the ARM] I was cheering them on because it wasn't something I wanted to do. But I did leave them with the idea of, "Look, if I can do it here … there are two of you; there's one of me." TI: The 6502 and '816 are often found today in other forms, either as the physical core of a system-on-a-chip, or running on an FPGA. What are some of the latest developments? BM: I'm excited about what's going on right now. It's more exciting than ever. I was just given these flexible 6502s printed with thin films by PragmatIC! Our chips are in IoT devices, and we have new educational boards coming out. TI: Why do you think the original 65x series is still popular, especially among people building their own personal computers? BM: There is a love for this little processor that's undeniable. And the reason is we packed it with love while we were designing it. We knew what we were doing. Rod and I knew from our previous experience with the Olivetti CPU and other chips. And from my work with I/O chips, I knew [how computers were used] in the real world. People want to work with the 65x chips because they are accessible. You can trust the technology.
  • Spot’s 3.0 Update Adds Increased Autonomy, New Door Tricks
    Sep 15, 2021 03:32 PM PDT
    While Boston Dynamics' Atlas humanoid spends its time learning how to dance and do parkour, the company's Spot quadruped is quietly getting much better at doing useful, valuable tasks in commercial environments. Solving tasks like dynamic path planning and door manipulation in a way that's robust enough that someone can buy your robot and not regret it is, I would argue, just as difficult (if not more difficult) as getting a robot to do a backflip. With a short blog post today, Boston Dynamics is announcing Spot Release 3.0, representing more than a year of software improvements over Release 2.0 that we covered back in May of 2020. The highlights of Release 3.0 include autonomous dynamic replanning, cloud integration, some clever camera tricks, and a new ability to handle push-bar doors, and earlier today, we spoke with Spot Chief Engineer at Boston Dynamics Zachary Jackowski to learn more about what Spot's been up to. Here are some highlights from Spot's Release 3.0 software upgrade today, lifted from this blog post which has the entire list: Mission planning: Save time by selecting which inspection actions you want Spot to perform, and it will take the shortest path to collect your data. Dynamic replanning: Don't miss inspections due to changes on site. Spot will replan around blocked paths to make sure you get the data you need. Repeatable image capture: Capture the same image from the same angle every time with scene-based camera alignment for the Spot CAM+ pan-tilt-zoom (PTZ) camera. Cloud-compatible: Connect Spot to AWS, Azure, IBM Maximo, and other systems with existing or easy-to-build integrations. Manipulation: Remotely operate the Spot Arm with ease through rear Spot CAM integration and split-screen view. Arm improvements also include added functionality for push-bar doors, revamped grasping UX, and updated SDK. Sounds: Keep trained bystanders aware of Spot with configurable warning sounds. The focus here is not just making Spot more autonomous, but making Spot more autonomous in some very specific ways that are targeted towards commercial usefulness. It's tempting to look at this stuff and say that it doesn't represent any massive new capabilities. But remember that Spot is a product, and its job is to make money, which is an enormous challenge for any robot, much less a relatively expensive quadruped. For more details on the new release and a general update about Spot, we spoke with Zachary Jackowski, Spot Chief Engineer at Boston Dynamics. IEEE Spectrum: So what's new with Spot 3.0, and why is this release important? Zachary Jackowski: We've been focusing heavily on flexible autonomy that really works for our industrial customers. The thing that may not quite come through in the blog post is how iceberg-y making autonomy work on real customer sites is. Our blog post has some bullet points about "dynamic replanning" in maybe 20 words, but in doing that, we actually reengineered almost our entire autonomy system based on the failure modes of what we were seeing on our customer sites. The biggest thing that changed is that previously, our robot mission paradigm was a linear mission where you would take the robot around your site and record a path. Obviously, that was a little bit fragile on complex sites—if you're on a construction site and someone puts a pallet in your path, you can't follow that path anymore. So we ended up engineering our autonomy system to do building scale mapping, which is a big part of why we're calling it Spot 3.0. This is state-of-the-art from an academic perspective, except that it's volume shipping in a real product, which to me represents a little bit of our insanity. And one super cool technical nugget in this release is that we have a powerful pan/tilt/zoom camera on the robot that our customers use to take images of gauges and panels. We've added scene-based alignment and also computer vision model-based alignment so that the robot can capture the images from the same perspective, every time, perfectly framed. In pictures of the robot, you can see that there's this crash cage around the camera, but the image alignment stuff actually does inverse kinematics to command the robot's body to shift a little bit if the cage is including anything important in the frame. When Spot is dynamically replanning around obstacles, how much flexibility does it have in where it goes? There are a bunch of tricks to figuring out when to give up on a blocked path, and then it's very simple run of the mill route planning within an existing map. One of the really big design points of our system, which we spent a lot of time talking about during the design phase, is that it turns out in these high value facilities people really value predictability. So it's not desired that the robot starts wandering around trying to find its way somewhere. Do you think that over time, your customers will begin to trust the robot with more autonomy and less predictability? I think so, but there's a lot of trust to be built there. Our customers have to see the robot to do the job well for a significant amount of time, and that will come. Can you talk a bit more about trying to do state-of-the-art work on a robot that's being deployed commercially? I can tell you about how big the gap is. When we talk about features like this, our engineers are like, "oh yeah I could read this paper and pull this algorithm and code something up over a weekend and see it work." It's easy to get a feature to work once, make a really cool GIF, and post it to the engineering group chat room. But if you take a look at what it takes to actually ship a feature at product-level, we're talking person-years to have it reach the level of quality that someone is accustomed to buying an iPhone and just having it work perfectly all the time. You have to write all the code to product standards, implement all your tests, and get everything right there, and then you also have to visit a lot of customers, because the thing that's different about mobile robotics as a product is that it's all about how the system responds to environments that it hasn't seen before. The blog post calls Spot 3.0 "A Sensing Solution for the Real World." What is the real world for Spot at this point, and how will that change going forward? For Spot, 'real world' means power plants, electrical switch yards, chemical plants, breweries, automotive plants, and other living and breathing industrial facilities that have never considered the fact that a robot might one day be walking around in them. It's indoors, it's outdoors, in the dark and in direct sunlight. When you're talking about the geometric aspect of sites, that complexity we're getting pretty comfortable with. I think the frontiers of complexity for us are things like, how do you work in a busy place with lots of untrained humans moving through it—that's an area where we're investing a lot, but it's going to be a big hill to climb and it'll take a little while before we're really comfortable in environments like that. Functional safety, certified person detectors, all that good stuff, that's a really juicy unsolved field. Spot can now open push-bar doors, which seems like an easier problem than doors with handles, which Spot learned to open a while ago. Why'd you start with door handles first? Push-bar doors is an easier problem! But being engineers, we did the harder problem first, because we wanted to get it done.
  • Will iPhone 13 Trigger Headaches and Nausea?
    Sep 15, 2021 07:25 AM PDT
    Tim Cook is "so excited for iPhone 13." I'm not, because yet again, Apple's latest and greatest tech sits behind an OLED display. And OLEDs, for some of us, cause nausea, headaches, or worse. I explain why Apple's OLED displays, that dim by flickering on and off rather than by voltage adjustments, trigger health issues here. The iPhone 13 series, launched Tuesday, has cool video features, like automatically changing focus on the fly. The phones have longer battery lives. They have better processors. But it doesn't come with an LCD option, the second generation that's OLED only. Watching the livestream of the iPhone 13 intro event this week, I had a moment of hope, albeit one that could be a little hard on the budget. The OLED screens on the iPhone 13 Pro models (starting at $999 for the Pro, $1099 for the Pro Max) sport a refresh rate of 120 Hz, instead of 60 Hz of other models. The rate of the flicker—the pulse width modulation (PWM) is typically four times the refresh rate, and the slower the flicker the worse the effects on the sensitive, so a higher refresh rate could potentially translate to higher frequency PWM, and trigger problems in fewer people. However, these new screens aren't designed to always run at 120 Hz. They will adjust their refresh rate depending on the content, Apple's executives explained, with movies and games running at the highest speed and more static things like photos and email at far slower rates, as low as 10 Hz. (Reducing the refresh rate extends battery life.) So it's hard to say whether this new display is better or worse for the motion sensitive. It's possible that Apple will offer a user option to lock the refresh rate at 120 Hz in spite of the hit on battery life, no word yet from Apple on that, and I won't really know if that will help unless I try it. Will my motion sensitivity force me to fall further and further behind as Apple's phone technology advances? Apple's September announcements did suggest a possible path. Perhaps my next phone shouldn't be a phone, but rather an iPad Mini. I'd have to back off on a few things I consider essential in a phone—that I could hold it in one hand comfortably and fit it in my back jeans pocket; at 5.3 by 7.69 inches the Mini is a little big for that. But Apple's new mini packs in much of the same technologies as its top-of-the-line iPhone 13s—the A15 Bionic chip, Center Stage software to automatically keep the subjects in the screen during video calls, and 5G communications, all behind an LCD, not an OLED, display. And oooh, that wisteria purple!
  • Competing Visions Underpin China’s Quantum Computer Race
    Sep 15, 2021 07:20 AM PDT
    China and the US are in a race to conquer quantum computing, which promises to unleash the potential of artificial intelligence and give the owner all-seeing, code-breaking powers. But there is a race within China itself among companies trying to dominate the space, led by tech giants Alibaba and Baidu. Like their competitors IBM, Google, Honeywell, and D-Wave, both Chinese companies profess to be developing "full stack" quantum businesses, offering access to quantum computing through the cloud coupled with their own suite of algorithms, software, and consulting services. Alibaba is building solutions for specific kinds of hardware, as IBM, Google, and Honeywell are doing. (IBM's software stack will also support trapped ion hardware, but the company's focus is on supporting its superconducting quantum computers. Honeywell's software partner, Cambridge Quantum, is hardware agnostic, but the two companies' cooperation is focused on Honeywell's trapped ion computer.) Baidu is different in that it is building a hardware-agnostic software stack that can plug into any quantum hardware, whether that hardware uses a superconducting substrate, nuclear magnetic resonance, or ion traps to control its qubits. "Currently we don't do hardware directly, but develop the hardware interface," Runyao Duan, Baidu's head of quantum computing, told the 24th Annual Conference on Quantum Information Processing earlier this year. "This is a very flexible strategy and ensures that we will be open for all hardware providers." Quantum computers calculate using the probability that an array of entangled quantum particles is in a particular state at any point in time. Maintaining and manipulating the fragile particles is itself a difficult problem that has yet to be solved at scale. Quantum computers today consist of fewer than 100 qubits, though hardware leader IBM has a goal of reaching 1,000 qubits by 2023. But an equally thorny problem is how to use those qubits once they exist. "We can build a qubit. We can manipulate a qubit and we can read a qubit," said Mattia Fiorentini, head of machine learning and quantum algorithms at Cambridge Quantum in London. "The question is, how do you build software that can really benefit from all that information processing power?" Scientists around the world are working on ways to program quantum computers that are useful and generalized and that engineers can use pretty much straight out of the box. Of course, real large-scale quantum computing remains a relatively distant dream—currently quantum cloud services are primarily used for simulations of quantum computing using classical computers, although some are using small quantum systems—and so it's too early to say whether Baidu's strategy will pay off. “We can build a qubit. We can read a qubit. But how do you build software that can really benefit from all that information processing power?" In the past, Alibaba worked with the University of Science and Technology of China in Hefei, the capital of central China's Anhui province, which currently has the world's most advanced quantum computer, dubbed the Zuchongzhi 2.1, after China's famous fifth century astronomer who first calculated pi to six decimal places. The company is also building quantum computing hardware of its own. China's most important quantum scientist, Pan Janwei, also worked for Alibaba as scientific advisor. Earlier this year, Pan's team set a new milestone in quantum computation with the 66-qubit Zuchongzhi 2.1. Pan and his team ran a calculation on the device in about an hour and a half, which would take the world's fastest supercomputer an estimated eight years to complete. Baidu, meanwhile, has been releasing a series of platforms and tools that it hopes will put it ahead when quantum computers eventually become large enough and stable enough to be practical. Last year, it announced a new cloud-based quantum computing platform called Quantum Leaf, which it bills as the first cloud-native quantum computing platform in China—a bit of semantics apparently intended to put it ahead of Alibaba's cloud division, which began offering a cloud-based quantum platform with the Chinese Academy of Sciences several years ago. Unlike Alibaba's platform, Quantum Leaf's cloud programming environment provides quantum-infrastructure-as-a-service. Baidu's cloud-native quantum computing platform Quantum Leaf provides access to the superconducting quantum processing unit from the Institute of Physics, Chinese Academy of Sciences. Baidu also released Paddle Quantum, a device-independent platform for building and training quantum neural network models for advanced quantum computing applications. It combines AI and quantum computing using the company's deep learning framework called PaddlePaddle—Paddle means PArallel, Distributed, Deep Learning—which has 3.6 million developers and can support hyperscale training models with trillions of parameters. Paddle Quantum, in turn, can be used to develop quantum neural network models for software solutions. Users can then deploy those models on both quantum processing units or simulators through Quantum Leaf. Baidu's quantum activities are largely focused on quantum artificial intelligence, an extension of Baidu's current artificial intelligence activities. Baidu also offers a "cloud-based quantum pulse computing service" called Quanlse, intended to bridge the gap between hardware and software through sequences of pulses that can control quantum hardware and reduce quantum error, one of the biggest challenges in quantum computing. "We see an increasing number of demands from universities and companies to use our quantum platform and collaborat[e] on quantum solutions, [which] is an essential part of our quantum ecology," a Baidu spokesperson said. Baidu's quantum activities are largely focused on quantum artificial intelligence, an extension of Baidu's current artificial intelligence activities. Quantum computing is expected to accelerate the development of artificial intelligence both by making models faster but also by allowing compute-intensive models not currently possible on classical computers. The company established a quantum computing institute in 2018 whose research includes classification of quantum data, which opens the door to quantum machine learning. To classify chemical compounds as toxic or non-toxic, for example, data scientists currently use classical means. But because the underlying data—the molecules and their configurations—is quantum data, it would be faster and more accurate to classify that quantum data directly with a quantum computer. Quantum information is encoded in the probability distribution of qubit states. That probability distribution is reconstructed by collecting samples with classical means, but the number of samples needed grows exponentially as you add qubits. "The more you add qubits to your quantum system, the more powerful the system, but the more samples you need to take to extract all useful information," says Cambridge Quantum's Fiorentini. Existing methods for quantum classification are impractical because hardware and infrastructure limitations restrict the complexity of the datasets that can be applied. Baidu researchers' new hybrid quantum-classical framework for supervised quantum learning uses what they call the “shadows" of quantum data as a subroutine to extract significant features—where “shadows" here refers to a method for approximating classical descriptions of a quantum state using relatively few measurements of the state. "If we can get all the key information out of the quantum computer with a very small number of samples without sacrificing information, that's significant," says Fiorentini. Baidu's hybrid quantum-classical framework, meanwhile, sharply reduces the number of parameters, making quantum machine learning models training easier and less compute intensive. In the near term, the company says, Baidu is pursuing more efficient and more powerful classical computing resources that can accelerate its AI applications, from training large-scale models to inferencing on the cloud or edge. In 2018, it developed a cross-architecture AI chip called Kunlun, named or the mountain range on the Tibetan plateau that is the mythological origin of Chinese civilization. Baidu has produced more than 20,000 14-nm Kunlun chips for use in its search engine, Baidu AI cloud and other applications. It recently announced the mass production of Kunlun II, which offers 2-3 times better performance than the previous generation, using the world's leading 7nm process and built on Baidu's own second-generation cross-platform architecture. Kunlun II has a lower peak power consumption while offering significantly better performance in AI training and inferencing. The chip can be applied in multiple scenarios, including in the cloud, on terminal, and at the edge, powering high-performance computer clusters used in biocomputing and autonomous driving.
  • Electric Motor Enables Chain-Free Bike-by-Wire
    Sep 15, 2021 06:59 AM PDT
    An increasingly-seen sight in Berlin and other German cities is the oversized electric cargo delivery bike, hissing along (and sometimes in bike lanes) like parcel-laden sailboats on appointed Amazon rounds. German manufacturer Schaeffler sees an opportunity: it is introducing a new generator at the heart of a smart drivetrain concept that some observers are calling bike-by-wire. It's a bike with no chain. Schaeffler's e-motor assembly was among the more out-of-the-ordinary items on display at the recent IAA Mobility show in Munich, which used to be the Frankfurt Motor Show, and more accustomed to roaring supercars and sleek news Benzes (and a thronging public, in pre-Covid times). But in some ways Schaeffler's pedal-cranked generator looked familiar; it's the world around it that's changing. That just might include reimagining the 130-year-old chain-driven bicycle. Schaeffler is working with German electric drive maker and systems integrator Heinzmann to develop a complete bike-by-wire drivetrain. The partners had a prototype on display in Munich (and the previous week at Eurobike) with a robust cargo three-wheel e-bike made by Bayk. Production models could come out as soon as first-quarter 2022, says Marc Hector, an engineer in Schaeffler's actuator systems unit and one of the developers on the pedal generator project. It's a hard thing to beat pedal-turns-sprocket. But maybe conditions are changing. Bike by wire physically de-links two kinetic systems: the turning pedals and the powering wheel on a bike. They are instead linked by a controller, an electronic brain, which directs power to either battery or hub motor. It also sends a resistance signal to the pedal, so the rider feels that he or she is pushing against something. Instead of producing motion, pedaling is producing current. Taking the chain out of the mix—if done successfully—would fly open the cramped world of cycle design to new shapes and configurations. Remove the electronic brain, however, and you're left with a stationary exercise bike bolted to a wheeled frame powered by rear electric motors. No wonder industrial designers and engineers have toiled for years on the concept: it's a hard thing to beat pedal-turns-sprocket. But maybe conditions are changing. Schaeffler's pedal-powered generator enables new, chainless e-bike designsSchaeffler Schaeffler is an auto parts and industrial manufacturer which made its name as a ball-bearing and clutch maker. It's developed electro-mobility products for 20 years, but has been on a buying spree: snapping up an engineering specialist firm in e-drives and another in the winding technologies used, among other things, to superefficiently wrap copper wire inside electric motors. It launched a new e-mobility business division that, reports Automotive News Europe, includes 48-volt auto powertrains as well as subsystems for plug-ins and full-electric vehicles. Here it's a different scale of electrics: Schaeffler's pedal generator is a self-contained four-kilo crank-driven e-machine in a cylindrical housing the shape of an oversized soup can placed in the bottom bracket of a cargo bike. The pedals turn the crank running through a standard brushless DC machine inside: fixed coil copper windings around an iron core are arranged within the cylinder as the generator stator. Magnets in the turning rotor create the current. Temperature sensors and a controller are housed along with the generator. The bike-by-wire controllers direct that current where needed: to the onboard battery for charging, to the interface display, to the rear hub traction motors that propel the bike, and back to the rider's feet in the form of counter-torque, giving the feeling of resistance when pedaling. The trick will be by synching it all up via Controller Area Network (CAN) bus, a 500 kbits/sec messaging standard which simplifies the amount of cabling needed. It should move the bike on one hand, and independently send the "chain feeling" back to the rider. Move pedal, move bike. “Pedal by wire has huge potential. Micromobility is coming." "The feeling should be the same as when there is a chain there," says Thorsten Weyer, a Schaeffler system architect. "But there is no chain." Propelling the bike will be the two Heinzmann hub motors, which the controller can get rolling set at European Union specs at 125 watts of power each, 250 total (500 watts in mountainous Switzerland, 600 in Austria). Each hub can each generate 113 newton-meters of torque on the axle, powering it ahead. "With the hub motor you have power where you need it," says Heinzmann electric drives managing director Peter Mérimèche. The controller's programmed with nine gear settings: the countercurrent controlling torque on the axle is reduced or increased automatically based on the grade the bike is traveling on. Designers have dreamed of chainless bikes for more than a century—in analogue form—and at least 25 years for e-bikes, as Andreas Fuchs, a Swiss physicist and engineer, developed his first chainless working models in the mid-90s. Challenges remain. Han Goes, a Dutch consultant and bicycle designer, worked with a Korean auto supplier a decade ago on a personal portable chainless folding bike. Pedaling parameters proved a struggle. "The man and the machine, the cyclist and the generator, the motor: nothing should feel disruptive," he says. If so, the rider feels out of step. "It is like you are pedaling somewhere into empty space." Goes is still at it, working with design partners on a new chainless cargo bike. Our parcels keep needing delivery, and the city is changing. "Pedal by wire has huge potential. Micromobility is coming," he says. Dutch and Danish and other developers are at it, too. "It offers design and engineering freedom. Simplicity. Less parts and maintenance. Traditional chain drives can never offer that."

Engineering on Twitter