Arrow

What went into creating Vincent, a hyper-real digital human

21 January, 2020
What went into creating Vincent, a hyper-real digital human | News | Pause Awards
George Hedon, founder and CEO, Pause Fest & Awards
George Hedon
21 January, 2020
Share

Until very recently, directly mapping an actor’s performance onto photorealistic digital humans using real-time rendering was considered impossible. We’d seen the likes of Avatar and Alita in the movies. Still, these involved time-consuming offline rendering and post-processing work.

But thanks to the advancements in graphics hardware and software, as well as the relentless hard work from innovative teams in the field, we’re seeing real-time rendered digital humans. Meet Vincent, a digital human born from Korea-based creative studio, Giantstep.

Creating Vincent was not an easy feat. After much deliberation and research into potential joint partnerships, the studio’s R&D arm, GXLab took on the challenge to develop the technology in-house – with just five artists. To make Vincent a reality, the team quickly identified three key challenges they needed to overcome.

Vincent character render

Skin and hair: the visual technology challenge

The first technical issue the team faced lay in ensuring that they had access to the necessary shading technology. Developing skin and hair features can cost an exorbitant amount of time, manpower and cost. With that in mind, the team turned to Unreal Engine. “Unreal Engine’s material editor and its powerful skin shading features like Transmission and Dual Lobe Specularity played a major role in boosting Vincent’s skin quality up to offline rendering levels without any additional development,” said Sungku Kang, Giantstep’s director of research and development.

The team made good use of Unreal Engine’s online learning courses to get to grips with the technology’s development processes and properties. “By leveraging all the available information, we were able to accurately understand how changing different parameters would affect the outcome. And use that information with more precise intent, rather than entering random numbers and leaving the results to chance,” Kang said. “Also, using material instancing to make immediate parameter changes and preview the results was very useful. This dramatically reduced the time spent in the final fine-tuning stage.”

The Meet Mike free asset sample provided by Epic Games also gave the team hints about the technologies they’d require to create the visuals for Vincent. Exploring the asset, they gathered necessary information, including how to express fine facial hairs and what kind of data is required for the final hair shape. “With all this information, our developers were able to reduce development time by clearly setting objectives for Maya plugins or scripts,” Kang explained.

Vincent character render

Creating lifelike facial expressions in real-time

Choosing and implementing technologies for effective facial expressions was another challenge the team had to overcome. They found that most solutions were developed for offline rendering by default and were of low, video-game quality – which limited the level of detail or the ability to customise. They started researching the technology that was the nearest fit, and then based on their findings, they developed their own solution in-house.

Their first task was to assess which candidate offered an accurate three-dimensional location data while providing a high degree of freedom for the data format. They settled on Vicon Cara, a head-mounted camera apparatus, as the facial animation capture system that best met their needs. Using this device, marker locations can be set with great flexibility, and they can then be translated into three-dimensional data with extremely high accuracy.

“Most solutions at that time only scanned two-dimensional location information data of facial landmarks. The fact that Cara was able to capture three-dimensional data made it a good choice,” says Kang. However, Cara is for offline rendering, making real-time data transfer impossible out of the box. To resolve this, we decided to create a neural network that uses deep learning to infer 3D marker locations from 2D images.”

First, the team added one additional camera to Cara. When capturing the facial movements of the actor, video from the other camera was separately saved and used as the learning data. From this, the developers created artificial intelligence that can infer the 3D marker locations with high accuracy from a 2D image input. Machine learning was further leveraged in as many areas as possible, including for emphasising facial expressions and setting up blend shape weight. “Giantstep gained lots of experience in machine learning through this process. We were encouraged by the fact that the effective use of machine learning can help a small team overcome its limitations,” Kang said.

Vincent character render

Optimising the digital human pipeline

The last major technical hurdle the team faced was in securing technology for optimising the pipeline. Given the small size of the team, they knew right away that efficiency was crucial. The most important initiative was to reduce the burden of manual iteration and to facilitate automation as much as possible. “This is another major reason why Unreal Engine was the best choice,” says Kang. “Unreal Engine’s support for Python and the convenience that brings to creating plugins allowed us to easily resolve iteration issues as well as to easily develop relevant tools.”

A case in point is the work the team did on photorealistic facial expressions. This process involved trying many combinations for the shape and number of areas into which the face is divided. And then followed by a preview of the results as quickly as possible. Because changing the facial areas required textures to be newly combined, assets had to be re-imported, and material composition details had to be readjusted. “If the whole process were to be manually handled, an artist would have to spend a full day on simple iterations to preview the results,” says Kang.

Instead, using Python scripting in Unreal Engine, the team was able to automate the management process of laborious tasks such as importing assets. “Tasks which previously required a full day whenever the data changed were automatically completed in just a few minutes,” continues Kang. “By maximising automation through various Maya plugins and scripts, and developing Unreal Engine plugins, Project Vincent was completed in a short time frame, despite the small size of the team.”

Vincent character render

Beyond movies: the future of real-time digital humans

Real-time digital human development technology is expected to change the production of entertainment media that traditionally relied on offline rendering technology. It’s even predicted to move into other industries. 

Synced with AI speakers or assistants, the technology could enable users to experience more intuitive AI services. We could start to see high-quality, hyper-real character creation used for effective cutting-edge marketing and promotional activities. 

Giantstep intends to be a key part of this story, committed to advancing their technology and with plans to reveal its next development results as soon as possible.

Video by Giantstep

Let Pause Awards tell your story
Entering first-timers
Check 2023 Pause Awards winners
Winners
Pause Awards Night: the community of champs
Pause Awards Night
2023 Years Summary page is here
Summary Page

Related posts


See who won in 2023


Read more

Category tracks



The Culture track seeks entries that exemplify the development of an inclusive and collaborative workplace culture, crucial for organisational success and growth. Submissions should illustrate transformative strategies implemented across core business functions, reflecting expertise, leadership, and a clear vision. These strategies should demonstrate a positive impact on the business, its employees, stakeholders, and customers, showcasing how a nurturing culture contributes to overall growth and innovation.

The Excellence track celebrates businesses that have demonstrated remarkable success and impact within their industry, regardless of their size. Entries for this category should illustrate how the business's strategic execution, guided by visionary leadership and exceptional expertise, has significantly influenced the industry. The focus will be on the tangible outcomes of these strategies, highlighting the company's ability to innovate, lead, and leave a lasting impression in its field.

The Good track is focused on recognising companies dedicated to their mission of creating a positive societal impact. Entries should highlight innovative solutions aimed at addressing global challenges such as environmental sustainability, social equity, and community well-being. Successful submissions will demonstrate how their initiatives contribute to significant changes, tackling issues like gender inequality, social imbalance, and injustice, thereby setting a precedent for responsible and impactful business practices.

Awarded to a single entry in a category, Grand Prix is the top accolade in that category. To win the Grand Prix, the entire Judging Board must unanimously agree that the entry deserves the accolade of being the best in that category.

The Growth track is about showcasing how businesses have enhanced productivity and execution to make a significant impact in their market. Entries should highlight innovative strategies that have optimised the customer journey, leading to increased engagement and commercial success. This track is for those who have effectively used their understanding of the market and customer needs to scale their business and achieve notable growth.

The Operators track seeks entries that showcase leadership in fostering innovation within their ecosystems. Ideal submissions will highlight how your initiatives have catalysed the creation of new products or services, bolstered by a culture that uplifts internal teams and the wider community. This track celebrates those who drive progress by empowering voices and forging transformative connections in their industry.


Categories


  • Always Evolving New

    Acknowledges a culture of continuous learning and improvement within a company, encourages employee development, and always looks for ways to optimise and innovate operations.


  • Big & Mighty Culture New

    Recognises companies (51-1000 employees) for their exceptional efforts in cultivating a culture that champions innovation.


  • Diverse Co.

    Celebrates workplaces dedicated to fostering innovation through a vibrant tapestry of diverse talents and perspectives.


  • Small & Mighty Culture New

    Applauds small teams of fewer than 50 employees for their remarkable ability to foster innovation and set new benchmarks in their domain.


  • Work Sweet Work

    Acknowledges companies that have achieved extraordinary employee engagement, satisfaction, and retention, creating an exemplary workplace culture.


  • Company of the Year

    Honours an outstanding company that not only showcases exemplary leadership and resilience but also demonstrates a clear, quantifiable impact over the past year.


  • Customer Obsessed New

    Applauds prioritisation, understanding and meeting the needs and expectations of their customers, including feedback, personalised services and ensuring high satisfaction.


  • Defiant Ones

    Recognises firms that lead technological innovation, setting new standards and defining the future of their industries.


  • Hammer

    Celebrates companies that have not only defined success on their own terms but have also concretely demonstrated their triumphs over the past year.


  • I Wish I'd Done That

    Applauds ingenious, simple solutions in niche markets that spark a universal sentiment of admiration.


  • On the Rise

    Celebrates early-stage startups, under 4 years, that have shown remarkable progress and success in their initial years.


  • Prodigy

    Recognises young visionaries under 30 with outstanding talents and achievements, setting them apart in their fields.


  • Singularity

    Highlights AI-driven companies that are pioneering new services or integrating AI to create innovative products.


  • Women in Business

    Celebrates female entrepreneurs and leaders who are forging paths for future generations by creating more equitable opportunities.


  • B-Good

    Recognises businesses that redefine commercial success by positively transforming our economic system for the benefit of all stakeholders.


  • Circular Pioneer

    Celebrates companies at the forefront of circular economy practices, making substantial and sustainable differences.


  • Ethical Leadership New

    Commends companies exemplifying leadership in environmental and social governance, setting standards in corporate responsibility and ethical practices.


  • Going Green

    Acknowledges companies leading large-scale sustainable initiatives, addressing critical areas like clean energy and environmental justice.


  • Collab New

    Honours strategic partnerships and alliances with other businesses to leverage complementary strengths, access new markets, and share resources for synergistic growth and innovation.


  • Design First

    Celebrates companies where design is the heartbeat of innovation, engaging the whole team in crafting groundbreaking products.


  • Going Global New

    Applauds companies that have expanded their reach globally, showcasing exceptional growth and international success.


  • Great Pivot

    Acknowledges companies that adeptly navigate change, leveraging insights to pivot towards delivering outstanding products.


  • Movement

    Recognises unforgettable event experiences that resonate emotionally, drive social change, or achieve viral status.


  • New Model

    Honours businesses revolutionising markets with elegant and disruptive business models.


  • Next Level New

    Applauds companies which had successful strategic acquisition and integration of another entity, staff or an asset that allowed them to further grow and scale their services.


  • Wildfire

    Celebrates companies that have rapidly and significantly impacted their marketplace, online and offline, setting new standards for success.


  • Angel

    Honours visionary angel investors who've significantly contributed to early-stage startups, guiding them toward success.


  • C-Suite

    Recognises exceptional leadership qualities in high-ranking executives, showcasing their management and interpersonal excellence.


  • Creative Maverick New

    Honours unconventional thinkers with exceptional abilities to execute typically unimaginable initiatives, expertly coordinate teams, secure funding or successfully launch multiple ventures.


  • Ecosystem Star

    Celebrates individuals who energise and nurture their communities, fostering engagement and advocacy.


  • Future Builder

    Recognises the determination and resilience of founders who've embarked on the challenging journey of starting their own ventures.


  • Quiet Achiever New

    Applauds the unsung heroes whose behind-the-scenes efforts have led to flawless project executions and exceeded expectations.


  • Splash Down Under New

    Acknowledges international companies making a noteworthy entrance into the Australian market, enriching the local business ecosystem.


  • TopCoder

    Recognises developers for their outstanding technical skills, clean code, and collaborative spirit.


  • Venture Cap

    Salutes venture capital firms with a track record of backing transformative businesses, contributing significantly to Australia's economic growth.


  • Visionary New

    Recognises effective leadership by setting a clear vision, motivating the team, and making strategic decisions that guide their business towards its goals.

Early Entries end on:
00d : 00h : 00m : 00s
Enter now