Microsoft Copilot’s Biggest GenAI Game-Changers

Also, Microsoft MVP Amy Babinchek tells CRN there are three big to-dos before any organization should put the tools to work: check permissions, scrub the data, and train the users.

The launch of Copilot, an AI layer that will soon be infused in nearly every one of Microsoft’s leading applications, is giving users power to control and process data on command, and it is promising to unleash an uptick in productivity that parallel’s the birth of the PC itself, said Microsoft CEO Satya Nadella.

“Copilot is one experience that runs across multiple surfaces,” he said at an unveiling event for Copilot this week in New York City. “It brings the right skills to you when you need them. It adapts to you. You no longer have to be the orchestrator of your apps, operating systems and devices. We are building Copilot into all of our most-used products and experiences and allowing you summon its power as a standalone app, as well as from Bing, on Edge, to Microsoft 365, and the newest versions of Windows.”

The event showcased dozens of capabilities across Microsoft products, and its Surface devices, that allow users to use natural language requests to write documents based on notes, documents, chats, and then rewrite them into emails, blogs, text messages, or any other form of written communication.

“So the context and intelligence of the web, your work data and what you’re doing in the moment, on your device are presented together in a single unified experience, centered around you, informed by your preferences and permissions,” he said on stage.

Amy Babinchak, the owner of Harbor Computer Services of Royal Oak, Mich., told CRN that there are three important to-dos before any organization adopts Copilot: check user permissions, scrub the data, and train the users.

“Businesses should engage in a data permissions audit before starting with Copilot,” she said. “It is very likely that they do not realize who has access to what and neither do the employees. It has access to everything the user has access to and will surface it when asked … Businesses are going to be surprised by how much that is. Guaranteed.”

Additionally, Babinchak said data cleansing will be a needed prior to the start of the implementation or Copilot’s results will suffer.

“Let’s say you ask it to perform some specification gathering for a new sales prospect. If Copilot has access to out-of-date information it might use it,” she said.

Training employees to use prompts will also be a critical skill if organizations are hoping to leverage the greatest benefits from Copilot.

“Being able to craft precise prompts which are paragraphs and paragraphs long to get the exact response and task performed in the way you are expecting, is a skill that no one has yet,” she said.

Babinchak said the organizations that invest in laying the groundwork for Copilot and gaining the in-house skills necessary to leverage it will be able to reap huge rewards.

“Those that invest in preparing their staff for this new technology are going to have a big advantage over others,” she said. The implications for productivity enhancement are enormous.”

Here are some of the biggest GenAI features of Copilot.

Windows 11

Windows 11 update will come with enhancements to 150 areas, and the infusion of AI into nearly all of the system’s standard apps. With a single click, a user can launch the AI companion, which is informed by what the user is doing on the PC at that time.

With the new copy and paste updates, as soon as users select text, photos, or other objects, their Copilot will begin offering ways to help in a separate panel. If it’s an email with instructions, it shows ways to reorganize the tasks more efficiently based on time or distance, or other parameters.

With the brand new feature, Ink Anywhere, users can use a stylus to input requests to Copilot, and generate responses from handwriting. If it’s a complex math problem, Microsoft’s vice president of product management, Carmen Zlateff, demonstrated that the software would solve the equation, and even show you its work.

“Since you can use your pen across apps like snipping tool, you can use the pen to snip a photo of the math problem,” Zlateff said during the demonstration. “You can see copilot immediately start solving just from that snip. And not only does Copilot solve the math problem, which feels magical, but trust me it’s real and powerful. But it explains how to solve it. And this serves as a great learning moment for all those future equations you’ll need to solve.”

Copilot on a users phone gives the device access to a users texts. In the demonstration, Zlateff showed how it can use data from the phone to find travel arrangements and make plans.

“I want to catch a play while I’m in New York on that trip. So I asked Copilot to write a text to my husband with the list of plays that are happening,” Zlateff said. “Copilot uses the information about the dates of my flight that it found on my phone. It also finds the plays going on during those dates. And they have available tickets and Copilot texts that to him right from my PC. Now that really feels like an assistant.”

Paint and Photos

Microsoft’s long-running Paint app was was remastered with AI as well. It will now include the ability to remove background objects, and comes with a preview version of Cocreator that Microsoft says brings generative AI to the drawing program once widely-associated with blocky pixel part. Paint also now has layers, which was introduced in Adobe Photoshop 3.0 in 1994.

The Photos app was also updated to make it easier to crop, tone, recolor, and orient your photos simply by asking. Additionally, users will be able to ask the app to remove background objects, and expand the boarders of the photo and have AI create an assumption of the background based on what it has observed.

Personalized Answers

Divya Kumar, Microsoft’s general manager of search and AI marketing, said its Bing Chat feature debuted in February. Since then, it has fielded 1 billion questions and that has given the company insight into how users can benefit.

“Modern search and check rarely take recent activity into consideration when generating a response to help you get the answers you care about faster,” she said. “We’re introducing personalized answers. This means your chant history can inform your results.”

In an example, using a highly general prompt about what to do for the weekend, the AI features took into account earlier questions about her dog to suggest pet friendly activities that were near her.

Microsoft Shopping

Kumar said the introduction of Copilot to shopping can guide novice users to make better purchases and ask better questions.

“My nephew loves playing soccer and I want to get him some new cleats. I’m not an expert so Copilot in Microsoft shopping will guide me to find the best pair by asking smart questions like his skill level, which is not something I would have thought to consider. Next it asks me to choose the type of field he’ll be playing on. I can even compare my options of top name brands from thousands of online merchants. I can get summarized reviews including the pros and cons of the shoe and co pilot gets me to a recommendation.”

In reverse it can take a picture to find a similar product for sale.

“It’s like having my own personal shopper,” Kumar said.

Image Generator

The company’s Bing image creator will undergo an overhaul, adding Dall-E 3 which fixes a common, if occasionally funny complaint of AI-generated pictures: human fingers.

“Even the little details like fingers and eyes and shadows are going to be dramatically improved over previous versions,” Kumar said.

Content credentials will label AI-generated art as such using “cryptographic methods,” Kumar said.

“Now every image generated is marked as created by AI using cryptographic methods, for better transparency.”

Bing Chat Enterprise - Multimodal Visual Search And Image Creator

Bing Enterprise Chat is able to coordinate several categories of information at once, to reprice the materials in a construction project with a user prompt, in one example from Microsoft. Then write a few paragraphs to clients about the changes, and place that in an email.

Additionally, using image recognition, users can ask Bing questions about objects in a picture.

“Now you can also search with images and create images in Bing chat enterprise with multimodal visual search and image creator,” Kumar said. “I grabbed a power converter before I left for the airport but I’m not sure it’s the kind that works in the UK. Fortunately, I can simply take a picture of the one I packed and ask if this will work in London. Bing uses GTP-4 and its image recognition capabilities to understand both the question and how it relates to the image to get the answer.”

M365 Chat

According to Collete Stallbaumer, general manager of M365, said Microsoft’s Chat feature now goes deeper than having conversation.

“You got a glimpse of M365 in March. But thanks to rapid advances over the past few months, today, we’re taking it to a whole new level, going beyond simple questions and answers to solve your most complex problems at work.”

Stallbaumer said 365 Chat will have a “deep contextual understanding” of the user’s job, their priorities, and organization.

“It’s like having someone attend all your meetings, even the ones you skip. Read all your emails, including the ones you missed. Know what is in every single file and recall it all at the speed of light,” she said.

Copilot can complete a complex multi-step process. It understands intent and cross references information from the web, with the data in your system, to connect the dots for users. She said Copilot can make recommendations based on constraints discussed.

“It not only understands the constraints, it thinks through the options to help you solve your problem,” Stallbaumer said.

By feeding the chat specific data, users can further refine their results.

“You point Copilot to the right files and ask it to pick the top five things and make them prominent,” Stallbaumer said.

Outlook - Sound Like Me

One issue with AI-generated emails is that they may not match the tone of the user, if the user were to actually be writing it. With the “Sound Like Me” feature, Copilot will match the user’s style and voice. It will use your sign off as well.

Designer

This is a new addition to M365 consumer apps, which creates and edits visual works for social media posts, invitations and more. It uses OpenAI’s Dall-E 3 as well as context from supplied documents, to propose images. It can expand a photo beyond borders, fill in or remove background objects, and generate images.

Designer will be incorporated in to M365 Copilot for consumers in Word, Microsoft said.

Surface

Microsoft released three new versions of the device.

The entry-level Surface Laptop Go 3 features a 12.4-inch display, an Intel Core i5 12th-generation processor and up to 15 hours of batter life. It will ship with Windows 11 with a starting price of $799.

The Surface Laptop Studio 2, with a haptic feedback touch pad that also provides adaptive touch for people without fingers or hands, is the most powerful laptop yet from the software maker. It boasts two times the graphic performance of the MacBook Pro M2 Max, Microsoft said. It features a 14.4-inch display and runs on an Intel 13th-generation i7 H class processor. The laptop, which will also ship with Windows 11, is priced starting at $1,999.

Both are available for pre-order now and expected to be released Oct. 3.

Microsoft also introduced the Surface Go 4, a tablet that runs on an Intel N200 processor and features a 10.5-inch touchscreen and up to 12.5 hours of battery life.