Arrow

What went into creating Vincent, a hyper-real digital human

21 January, 2020
What went into creating Vincent, a hyper-real digital human | News | Pause Awards
George Hedon
21 January, 2020
Share

Until very recently, directly mapping an actor’s performance onto photorealistic digital humans using real-time rendering was considered impossible. We’d seen the likes of Avatar and Alita in the movies. Still, these involved time-consuming offline rendering and post-processing work.

But thanks to the advancements in graphics hardware and software, as well as the relentless hard work from innovative teams in the field, we’re seeing real-time rendered digital humans. Meet Vincent, a digital human born from Korea-based creative studio, Giantstep.

Creating Vincent was not an easy feat. After much deliberation and research into potential joint partnerships, the studio’s R&D arm, GXLab took on the challenge to develop the technology in-house – with just five artists. To make Vincent a reality, the team quickly identified three key challenges they needed to overcome.

Vincent character render

Skin and hair: the visual technology challenge

The first technical issue the team faced lay in ensuring that they had access to the necessary shading technology. Developing skin and hair features can cost an exorbitant amount of time, manpower and cost. With that in mind, the team turned to Unreal Engine. “Unreal Engine’s material editor and its powerful skin shading features like Transmission and Dual Lobe Specularity played a major role in boosting Vincent’s skin quality up to offline rendering levels without any additional development,” said Sungku Kang, Giantstep’s director of research and development.

The team made good use of Unreal Engine’s online learning courses to get to grips with the technology’s development processes and properties. “By leveraging all the available information, we were able to accurately understand how changing different parameters would affect the outcome. And use that information with more precise intent, rather than entering random numbers and leaving the results to chance,” Kang said. “Also, using material instancing to make immediate parameter changes and preview the results was very useful. This dramatically reduced the time spent in the final fine-tuning stage.”

The Meet Mike free asset sample provided by Epic Games also gave the team hints about the technologies they’d require to create the visuals for Vincent. Exploring the asset, they gathered necessary information, including how to express fine facial hairs and what kind of data is required for the final hair shape. “With all this information, our developers were able to reduce development time by clearly setting objectives for Maya plugins or scripts,” Kang explained.

Vincent character render

Creating lifelike facial expressions in real-time

Choosing and implementing technologies for effective facial expressions was another challenge the team had to overcome. They found that most solutions were developed for offline rendering by default and were of low, video-game quality – which limited the level of detail or the ability to customise. They started researching the technology that was the nearest fit, and then based on their findings, they developed their own solution in-house.

Their first task was to assess which candidate offered an accurate three-dimensional location data while providing a high degree of freedom for the data format. They settled on Vicon Cara, a head-mounted camera apparatus, as the facial animation capture system that best met their needs. Using this device, marker locations can be set with great flexibility, and they can then be translated into three-dimensional data with extremely high accuracy.

“Most solutions at that time only scanned two-dimensional location information data of facial landmarks. The fact that Cara was able to capture three-dimensional data made it a good choice,” says Kang. However, Cara is for offline rendering, making real-time data transfer impossible out of the box. To resolve this, we decided to create a neural network that uses deep learning to infer 3D marker locations from 2D images.”

First, the team added one additional camera to Cara. When capturing the facial movements of the actor, video from the other camera was separately saved and used as the learning data. From this, the developers created artificial intelligence that can infer the 3D marker locations with high accuracy from a 2D image input. Machine learning was further leveraged in as many areas as possible, including for emphasising facial expressions and setting up blend shape weight. “Giantstep gained lots of experience in machine learning through this process. We were encouraged by the fact that the effective use of machine learning can help a small team overcome its limitations,” Kang said.

Vincent character render

Optimising the digital human pipeline

The last major technical hurdle the team faced was in securing technology for optimising the pipeline. Given the small size of the team, they knew right away that efficiency was crucial. The most important initiative was to reduce the burden of manual iteration and to facilitate automation as much as possible. “This is another major reason why Unreal Engine was the best choice,” says Kang. “Unreal Engine’s support for Python and the convenience that brings to creating plugins allowed us to easily resolve iteration issues as well as to easily develop relevant tools.”

A case in point is the work the team did on photorealistic facial expressions. This process involved trying many combinations for the shape and number of areas into which the face is divided. And then followed by a preview of the results as quickly as possible. Because changing the facial areas required textures to be newly combined, assets had to be re-imported, and material composition details had to be readjusted. “If the whole process were to be manually handled, an artist would have to spend a full day on simple iterations to preview the results,” says Kang.

Instead, using Python scripting in Unreal Engine, the team was able to automate the management process of laborious tasks such as importing assets. “Tasks which previously required a full day whenever the data changed were automatically completed in just a few minutes,” continues Kang. “By maximising automation through various Maya plugins and scripts, and developing Unreal Engine plugins, Project Vincent was completed in a short time frame, despite the small size of the team.”

Vincent character render

Beyond movies: the future of real-time digital humans

Real-time digital human development technology is expected to change the production of entertainment media that traditionally relied on offline rendering technology. It’s even predicted to move into other industries. 

Synced with AI speakers or assistants, the technology could enable users to experience more intuitive AI services. We could start to see high-quality, hyper-real character creation used for effective cutting-edge marketing and promotional activities. 

Giantstep intends to be a key part of this story, committed to advancing their technology and with plans to reveal its next development results as soon as possible.

Video by Giantstep

Pause Awards '22 entries are now open!
Website banner, enter awards
Easy to enter. Impossible to forget.
Have dreams? Go get them.
Early entries end on 15 July, at midnight.
For the most ambitius and forward-thinking people.

Applications are now open


Enter now

Category tracks



Creating a safe and open space in which everyone thrives and the collaboration flows through all levels and titles with the power of the one team. The entries should demonstrate how the transformation has been achieved across core business functions, that have delivered a positive impact and growth for business, staff, stakeholders and customers. Judges will only refer to written entries during the evaluation process. No additional research will be undertaken, and for this reason, we recommend you give detailed and sharp answers.

Celebrating commercial effectiveness through success in the marketplace by leaving a dent in the universe, no matter what the size of the company. The entries should demonstrate the highest levels of expertise, leadership and vision in the application of their business strategies. Judges will only refer to written entries during the evaluation process. No additional research will be undertaken, and for this reason, we recommend you give detailed and sharp answers.

Standing up for your purpose and going beyond together as a team aligned on the single mission to positively impact the environment, people and community. The entries should demonstrate ideas intended to change the world; work which sets out to positively impact ingrained gender inequality, social imbalance or injustice. Judges will only refer to written entries during the evaluation process. No additional research will be undertaken, and for this reason, we recommend you give detailed and sharp answers.

Only awarded to a single entry in each category track, Grand Prix is the highest accolade in Pause Awards. To win the Grand Prix, the entire Judging Board must unanimously agree that the entry deserves the accolade of being the best in class.

​​Focusing on the productivity and elegance of execution to deliver impactful ideas to a market (or many) with the power to cut through the noise. Entries should demonstrate how the innovation and optimisation of the customer journey led to scaled consumer engagement and commercial success. Judges will only refer to written entries during the evaluation process. No additional research will be undertaken, and for this reason, we recommend you give detailed and sharp answers.

Elevating every voice in the ecosystem is not just important for its collective success, industry development but also for the future growth. The entries should demonstrate ingenuity which leads to the creation of new products and services, through leadership, by elevating the internal culture and communities they serve. Judges will only refer to written entries during the evaluation process. No additional research will be undertaken, and for this reason, we recommend you give detailed and sharp answers.


Categories


  • Diverse Co. (NEW)

    For nurturing an inclusive environment. This award recognises a workplace that has successfully and intentionally encouraged innovation among diverse employees.


  • Large team culture (NEW)

    For fostering innovation. This award recognises a workplace with over 1000+ employees that has outpaced their cohort in fostering a culture of innovation.


  • Medium team culture (NEW)

    For fostering innovation. This award recognises a workplace with 21-999 employees that has outpaced their cohort in fostering a culture of innovation.


  • Small team culture (NEW)

    For fostering innovation. This award recognises a workplace with less than 20 employees that has outpaced their cohort in fostering a culture of innovation


  • Work sweet work

    For a great workplace culture. This award recognises a company that has managed to improve or attain outstanding levels of employee engagement, retention and satisfaction.


  • Company of the Year (NEW)

    For the best company. This award recognises a company leadership, success, ambition to excel against all odds with the solid future development plans.


  • Defiant ones

    For new tech breakthroughs. This award recognises a company that defines the future of the industry they play in.


  • Hall of fame

    For a lifetime achievement. This award recognises a game-changer who paved the way by breaking new grounds and inventing new categories.


  • Hammer

    For nailing it. This award recognises a company's success – on their own terms – in the market.


  • I wish I'd done that

    For sheer brilliance. This award recognises a company's simple and clever idea in a niche market.


  • Indi

    For new and First Peoples. This award recognises a successful company with its founders from indigenous or ethnic heritage that gained success in the market.


  • On the rise

    For early stage mavericks. This award recognises an early stage startup that has done well in its first three years.


  • Prodigy

    For being exceptional. This award recognises a young person with exceptional qualities or abilities.


  • Women in business (NEW)

    For breaking the mould. This award recognises a successful female leader and entrepreneur who is paving the way forward for the future generations.


  • B-Good

    For a good business. This award recognises a company that has done well to change our economic system to positively impact all stakeholders through their commercial practice.


  • Circular Pioneer

    For making a difference. This award category recognises a company that is pioneering their circular efforts.


  • Going green

    For delivering impact at scale. This award recognises a company that is actively leading and delivering impact and scale through sustainable efforts in clean energy, climate change, healthy environment, social and economic justice etc.


  • You've got ethics

    For the leadership in environmental social governance. This award category recognises a company that is promoting accountability, good governance, leadership in Corporate Social Responsibility and solving societal problems.


  • Design first (NEW)

    For design lead products. This award recognises a company that is using design to drive product innovation with the entire company onboard.


  • Inception

    For visionary thinkers. This award recognises a person that came up with the transformative idea that can change the world for better.


  • Out of the garage

    For scaling globally. This award recognises a company's global growth, expansion, and success.


  • Unexpected twist

    For a great pivot. This award recognises a company that has taken an unexpected pivot in their business or career.


  • Wildfire (NEW)

    For conquering a marketplace. This award recognises a company that has made a significant impact on the marketplace in the shortest amount of time.


  • Backbone (NEW)

    For backing the future idea. This award recognises a VC firm with a portfolio of companies that have made a significant contribution to the growth and success of the AU economy.


  • Brave Angel (NEW)

    For early stage support. This award recognises an angel investor that has garnered a portfolio of companies and has a good track record of helping founders succeed.


  • C-Suite (NEW)

    For the best leadership. This award recognises the highest-ranking executive that has impeccable leadership, management and interpersonal skills.


  • Do you know who I am

    For the silent achiever. This award recognises a person that has been responsible for the success of their company from the backstage.


  • Future builder (NEW)

    For grit to start a new venture. This award recognises a founder at any level and life stage who has decided to start jumping the hoops of their own business.


  • Hottest hub (NEW)

    For the best community. This award recognises a co-working space that provides boutique experience, world class facilities with community management.


  • Launch pad (NEW)

    For activating the support systems. This award recognises an accelerator that has a good reputation in providing the best startup program with industry connections.


  • Super connector

    For being the community champ. This award recognises a person for a significant contribution to the success of their clients or the community.

Early entries close in
00d : 00h : 00m : 00s
How to enter