 |
|
|
|||||
AI models have a voracious appetite for data. Keeping up to date with information to present to users is a challenge. And so companies at the vanguard of AI appear to have hit on an answer: crawling the webconstantly. But website owners increasingly dont want to give AI firms free rein. So theyre regaining control by cracking down on crawlers. To do this, theyre using robots.txt, a file held on many websites that acts as a guide to how web crawlers are allowedor notto scrape their content. Originally designed as a signal to search engines as to whether a website wanted its pages to be indexed or not, it has gained increased importance in the AI era as some companies allegedly flout instructions.In a new study, Nicolas Steinacker-Olsztyn, a researcher at Saarland University and his colleagues analyzed how different websites treated robots.txtand whether there was a difference between sites measured as reputable versus not reputable, specifically in terms of whether or not they allowed crawling. For many AI companies, “It’s kind of a do now and ask for forgiveness later thing, Steinacker-Olsztyn says.In the study, more than 4,000 sites were checked for their responses to 63 different AI-related user agents, including GPTBot, ClaudeBot, CCBot, and Google-Extendedall of which are used by AI companies in their effort to soak up information. The websites were then divided between reputable news outlets or misinformation sites, using ratings devised by Media Bias/Fact Check, an organization that categorizes news sources depending on their credibility and the factuality of their reporting. Across all 4,000 sites assessed, around 60% of those deemed to be reputable news websites blocked at least one AI crawler from accessing their information; among misinformation sites, only 9.1% did so. The average reputable site blocks more than 15 different AI agents through its robots.txt file. Misinformation sites, by contrast, tend not to shut out the crawlers at all. The biggest takeaway is that the reputable news websites keep well up-to-date with the evolving ecosystem as it pertains to these major AI developers and their practices, Steinacker-Olsztyn says. Over time, the gap between those who are willing to let bots crawl their sites and those that arent is widening. From September 2023 to May 2025, the proportion of platforms locking out crawlers increased from 23% to 60%, while the share of sites peddling misinformation stayed flat, the study found. The result, Steinacker-Olsztyn says, is that less reputable content is being hoovered up by and then spat out of AI models used routinely by hundreds of millions of people. Increasingly these models are also being used simply for information retrieval, replacing traditionally used options such as search engines or Google, Steinacker-Olsztyn adds. The conundrum over legitimate data For AI models to stay up-to-date on current events, they are trained on reputable sites, which is exactly what these sites dont want. The war over copyright and access to training data between AI companies and news sites is increasingly spilling into courtsThe New York Timess lawsuit against OpenAI, the makers of ChatGPT, for example, carried on into last week. Those lawsuits are prompted by allegations that AI companies are illegally scraping data on news websites to act as regularly updated, ground-truth-based training data for the models powering their AI chatbots. In addition to litigating their disputes, reputable news websites are blocking AI crawlers. Thats good for their businesses and rights. But Steinacker-Olsztyn is concerned about the broader impact. If reputable news is increasingly making this information unavailable, then this gives reason to believe this can affect the reliability of these models, he explains. Going forward, this is changing the percentage of legitimate data that they have access to. In essence: It doesnt matter to an AI crawler whether its viewing The New York Times or a disinformation website run out of Hoboken. Theyre both training data, and if one is easier to access than the other, thats all that matters. Not everyone is quite so certain about the negative impact of blocking crawlers. Felix Simon, a research fellow in AI and digital news at the University of Oxford-based Reuters Institute for the Study of Journalism, says he wasnt surprised to learn that sites trafficking in misinformation would want to be crawled, whereas traditional publishers have an incentive at this point to prevent such scraping. Some of these traditional publishers, he adds, still allow some scraping for a plethora of reasons. Simon also cautions that just because misinformation sites are more likely to open their doors to AI crawlers, it doesnt necessarily mean that theyre polluting the information space as much as we may fear. AI developers filter and weigh data at various points of the system training process and at inference time, he says. One would hope that by the same means by which the authors have been able to identify untrustworthy websites, AI developers would be able to filter out such data.
Category:
E-Commerce
When the new year rolls around, many people will resolve to get in better shape. Last year, Americans poured $44.8 billion into the fitness industry, flocking to gyms and buying at-home fitness equipment. But it usually takes just two weeks for people to abandon their goals. Gym memberships go unused. Peloton bikes collect dust. Researchers at the National Institutes of Health have found that amidst all the fitness options on the market, personal training tends to lead to better results for several reasons: It involves a personalized program, fits into the participant’s schedule, and requires being accountable to the trainer. But personal training is expensive, priced anywhere from $50 to $150 (or more) per hour for one-on-one sessions. Ray, a new AI-powered fitness app, wants to offer all the benefits of a personal trainer for a fraction of the price. (In fact, the service is free for early users, but will eventually start charging a monthly fee.) When you open the app, an AI trainer that looks and sounds like a real person will guide you through a workout. The program will be customized to your goals, your bodily limitations, the equipment you have handy, and the amount of time you have. And like a human trainer, Ray will observe your movements to help you improve your form and offer more challenging workouts as you improve. Ray will also follow up with you by text, like a real trainer, to see how the session went and to help schedule the next one. Importantly, the technology is going to keep improving as more people use Ray and as AI evolves. “The conversation’s going to get more fluid; the vision is going to get better,” says Colin Raney, Ray’s cofounder. “We’re currently working on grunt recognition, to see if we can identify how the user is doing based on the sounds they’re making.” [Photo: Ray] Why Personal Training Works Ray is the brainchild of Rich Miner, cofounder of Android, and Raney, an Ideo veteran and former CMO of PillPack. Over the years, both have relied on personal trainers and found them transformative. For Raney, it was clear that there were many aspects of working with a human being that could cultivate real behavior change. “I needed the accountability of not wanting to let my trainer down by not showing up,” he says. “Or tuning the workout to me if I had a bad day, or my back wasn’t feeling right, or if it looked like I was ready to push myself that day.” Raney has thought about improving people’s health by changing their behavior. He helped build PillPack, which was designed to help people actually take their medicine by delivering it monthly in packets sorted by date and time. He believed it would be possible to get people to workout more frequently if they had access to the qualities of a personal trainer. [Photo: Ray] “Our thesis was that if you build the right behavioral loop, people will workout more regularly,” he says. This aligns with research that finds that people who relied on a personal trainer lost fat and built muscle, with a lower rate of injury, compared to people who worked out alone or in groups. As Raney spoke with Miner about building a fitness tool, it became clear that AI technology was evolving to the point that it could mimic a personal trainer. Miner has been working on AI for decades, and has the patent to one of the first “wake words” for a voice-based personal assistant 35 years ago. “If you’re not a movie star or someone with a lot of money, you can’t afford to get that kind of personalized training,” he says. “But I realized that with agentic AI, you could actually start building virtual people who could watch you and talk to you naturally.” [Photo: Ray] Building an AI Native Tool The fitness industry is working hard to incorporate technology into existing tools. Over the last decade, there’s been an explosion of tech-enabled machines, from Peloton and NordicTrack machines with screens that provide feedback about the workout, to weight-lifting machines like Mirror, Tempo, and Tonal that can help count your reps. Now, these companies are figuring out ways to incorporate AI into their systems, to better tailor workouts to the user. Minor and Raney began building Ray two and a half years ago. What sets it apart from many other tools on the market is that it is built on AI, rather than simply retrofitting existing technology with AI. Ray is designed to approximate a real person that can interact with the user in a natural way. “It adapts to you,” Minor says. “You don’t have to change a bunch of settings to make sure the workout is tailored to you. You can just say, “Ray, my shoulder’s hurting today.” The technology is equipped with several cutting-edge AI technologies. It has natural language processing, to create real conversations with the user. It is able to observe the user across 35 different points, and has a machine learning algorithm that identifies your body movements. It is also equipped with an AI planner that helps you dynamically plan workouts based on the user and their workout history. Ray has also incorporated a lot of highly specific data about personal training. The data is trained on the textbooks and manuals that personal trainers use to get qualified. Raney also became certified as a personal trainer to ensure the Ray experience is as realistic as possible. “Ray’s team has a huge amount of domain expertise,” he says. “We have a lot of knowledge about things like what constitutes a good workout and how to create a complete workout in a given amount of time.” Raney believes that the seamlessness of the interaction is important because one big obstacle to behavioral change is decision fatigue. “Part of what holds people back is the mental load,” he says. “You have to decide when you will do the workout, and then make a lot of decisions about what exactly you’ll do and for how long. With a personal trainer, all you need to do is to show up at the agreed upon time and do the program.” Minor believes that consumers will immediately see the difference between an AI-native fitness app, versus one that is back-solving into an existing system. He compares it to how companies started making mobile-first apps instead of just adapting their websites to mobile apps. “That’s when you got Instagram and Uber,” he says. “People rethought what an app could be if you didn’t have to rely on a legacy application. That’s what we’re trying to do with Ray.” How An AI Personal Trainer is Different Six months ago, Ray quietly launched on the app store, without any marketing. Thousands of users have already started using it. The founders say they wanted to see how users interacted with it and use this data to further train the AI. When I tried it, I was impressed by how well the program adapted to my needs. In 20 minutes, I was able to do a range of exercises in my office without any equipment. As I did push-ups and jumping jacks, my Ray trainer was encouraging, telling me that I was on the right track. It also respected the fact that I hate burpees. (You can pick whether the trainer is supportive or a little more assertive, since different users respond to different approaches.) [Photo: Ray] But as with other agentic AI platforms I’ve used, I found that the interaction wasn’t perfectly seamless. I didn’t exactly feel like I was interacting with a human trainer; the AI trainer’s eyes seemed blank and unfocused. When I spoke, Ray would pause before responding to me in a way that made the conversation a little stilted. Ray’s founders say that these aspects of the interactions will only get better as the more people use the platform and the AI itself improves. But for now, Ray didn’t feel human enough that I felt bad about letting it down if I didn’t show up for a workoutthe way I wold if I were working with a real person. [Photo: Ray] Miner acknowledges that an AI agent won’t provoke the same level of accountability that a human would, but he says that there are still benefits to creating an anthropomorphic app. “It’s more than about creating a sense of guilt about letting a person down,” he says. “A trainer guides you through what to do and they’re watching you as you workout, counting your reps and motivating you. Ray gives you all of that.” And there are some ways that a virtual personal trainer is more convenient than a human one. You can do workouts at odd hours that a human may find objectionable. You don’t have to compete with the trainer’s other clients and you can cancel at the last minute. And then there’s the price. Right now, Ray is free. But in the coming months, the founders will develop a pricing structure that is designed to be significantly less than the price of hiring a human personal trainer. Over time, the founders believe that Ray will begin to feel even more like the real thing. “Ray has improved so much over the past six months since it’s launched,” says Raney. “It’s going to feel more and more real as time goes on.”
Category:
E-Commerce
A 220-pound, fully functional, solid-gold toiletonce offered to President Donald Trump as a satirical giftjust sold at a Sothebys auction for $12.1 million. The commode is a work of art called America created by Maurizio Cattelan in 2016. Cattelan is most well-known for his surreal, conversation-starting, and often controversial art concepts, like the 1999 piece La Nona Ora, which depicts a life-size Pope John Paul II getting struck by a meteorite, or the infamous 2019 piece Comedian, which is, put simply, a banana taped to a wall (which sold at auction for $6.2 million). After America debuted at the Guggenheim Museum in September 2016, it became an instant subject of public fascination, inspiring dozens of think pieces and even a front cover of the New York Post. In the nine years since, the intrigue surrounding the work has only grown after it was the target of a high-profile heist. Perhaps the most enduring legacy of America, though, will be its emergence as a striking symbol for Trumps first terma connection made even more poignant by the timing of its sale during Trumps second term, itself most visually recognizable by its glut of gilded motifs. The original America, on display at the Guggenheim Museum in 2016 [Photo: William Edwards/AFP/Getty Images] The storied, sometimes unbelievable history of America When it debuted to the public, America was not sequestered on a pedestal or inside a gallery space. Instead, it was located in the Guggenheims bathroom, where visitors were allowed five minutes each to, as one New York Post article put it at the time, crap all over America. In all, more than 100,000 guests lined up to do just that. At the time, the Guggenheim said that the artwork represented the American dream, with its utility ultimately reminding us of the inescapable physical realities of our shared humanity. Cattelan put it more bluntly to The New Yorker: Whatever you eat, a two-hundred-dollar lunch or a two-dollar hot dog, the results are the same, toilet-wise, he said. In 2019, the loo was sent to Englands Blenheim Palace, where it was to be put on view to the public for a second time. Just as the exhibition was set to open, a group of thieves broke into the palace, used sledgehammers and crowbars to pry the toilet out of the floor, and escaped in under five minutes. Two of the people involved in the theft were sentenced to prison in June of this year, but the original America has never been recovered. The America that just sold at Sothebys is actually a second version of the work, which Cattelan had previously alluded to but did not reveal in public until the sale. According to Sothebys, its the only other version of America in existence. People wait in line to use the fully functional installation of America at the Guggenheim Museum, 2016. [Photo: Christina Horsten/Picture Alliance/Getty Images] An 18-karat-gold throne In a new YouTube video about the whirlwind history of America, several Sothebys experts note that Cattelans reticence to provide much context about America is one of the reasons it became iconic. But the toilets connections to Trump were never exactly subtle. When America arrived at the Guggenheim just two months before Trump was elected in 2016, the museums then-blogger, Caitlin Dover, wrote in a post, The aesthetics of this throne recall nothing so much as the gilded excess of Trumps real-estate ventures and private residences. When Dover asked Cattelan about the connection, he said that while Trump wasnt top of mind when he conceived the piece, it was probably in the air. The allusion got a lot more explicit in 2018. That year, the Trump administration contacted the Guggenheim to request that the Vincent van Gogh painting Landscape With Snow be borrowed for Trumps private living quarters. According to The Washington Post, then-curator Nancy Spector, who wrote a book about Cattelans work in 1999, declined to lend the painting, suggesting America instead. The White House, it seems, did not respond to the subtle act of protest. A gilded symbol Trumps penchant for gold decor dates back decades. In 2004, he told reporters that the reign of Louis XIV represented his favorite styleand the gilded, rococo-esque aesthetic has become a visual hallmark of both his presidencies. This year alone, Trump has decked out the Oval Office in golden objects, accepted a luxury jet from Qatar filled with gold furniture, and begun construction on a new White House ballroom that will be, predictably, very gold. In some ways, a golden toilet feels like the ultimate symbol of a second term in which Trump has repeatedly strengthened his ties with billionaire investors while the average American is in the midst of a cost-of-living crisis. But when asked about the goal behind agreeing to offer America to the Trump family in 2018, Cattelan told The Post that his reasoning was a bit more lighthearted. “What’s the point of our life? he said at the time. Everything seems absurd until we die, and then it makes sense.” The original America at Blenheim Palace, 2019 [Photo: Leon Neal/Getty Images]
Category:
E-Commerce
All news |
||||||||||||||||||
|
||||||||||||||||||