Real-time WebGL video manipulation | by Szabolcs Damján

This is part III of the “Manipulating video in a browser” series. In the previous articles, we experimented with the new “Insertable Streams for ... GetunlimitedaccessOpeninappHomeNotificationsListsStoriesWritePublishedinDoclerEngineeringReal-timeWebGLvideomanipulationlive!ThisispartIIIofthe“Manipulatingvideoinabrowser”series.Inthepreviousarticles,weexperimentedwiththenew“InsertableStreamsforMediaStreamTrack”APItomodifythewebcam’svideostreambeforesendingitfurther—totheWebRTCmodule,forexample.ThefirstattemptusedJavaScripttoprocessthevideoframes:https://medium.com/docler-engineering/manipulating-video-in-a-browser-5b37f8149d9bThesecondoneusedWebAssemblytodothesameoperation:https://medium.com/docler-engineering/video-manipulation-with-webassembly-3477a0c8524dOurexampleisasimplegreenscreeneffectwheretheapplicationswapsthegreenbackgroundtoanicesummerscene.originalscenemodifiedsceneTheresultsweresomewhatdisappointing:alowerperformancePC,oranaveragemobiledevice,wasnotabletodeliverasolid30frame/secin720p(HD)!Raisethebet!Thereisahiddenworkhorseinnearlyeverycomputer,regardlessifit’sasmartphoneorahigh-endPC:thegraphicsprocessora.k.a.thefamousGPUNow,I’llshowhowtogettheGPUtoprocessvideoframesinabrowserenvironment.WebGLtotherescueInmodernbrowsers,ourkeytoGPUprocessingistheWebGLAPI.Let’sdustoffourpreviousexampleandswaptheframetransformerfunctionforWebGL’sone!GPUprocessingisalittlebitdifferentthanourpreviousexamples.WewillneedtoprepareanduploadspecialprogramstotheGPU.Withthehelpofthevertexandfragmentshaderprograms,wecantransformourvideoframes.EachframewillbeuploadedtotheGPUasatexture,thenthemodifiedframewillbepassedfurthertothevideostream.processflowImustmentionthatthisarticlewon’texplainallthedetailsofWebGL,butitwilldemonstrateaWebGLbasedimplementationofacertaintechnicalproblem.Ihaveaddedsomelinkstolearningmaterialintheappendixifyouarewishtodigdeeperintothetopic.PrepareforWebGLmagicTouseWebGL,youwillneedthefollowingcomponents:AHTMLcanvaselementItisnotnecessarytoattachthiselementtotheDOM,itcansimplyexistinmemory.ThiswillbetherunningcontextofyourWebGLprograms,andwillholdtheresultingimageaftereveryprocessinground.ThevertexshaderSinceourimageisasimple,staticrectangleinthe3DspaceprovidedbyWebGL,thiscomponentwillnotdotoomuch,onlycoordinatesystemtransformations.ThefragmentshaderThispieceofprogramwilldotheimagemanipulationbyrunningontheGPUdirectly.WebGLinitializationCheckoutthedetailsintheupcomingsections.WebGLinitializationBeforegettingtheGPUtodowhatwewant,weneedtoinitializetheWebGLsubsystem.Theinitializationcodeismostlygeneric,theonlyexceptionistheinterestingwayweneedtouploadtheYUVcolorformatvideoframes.ThefullinitializationscriptcanbefoundintheAppendix.Apartfromthebasicinitializationprocess,wealsoneedtouploadthebackgroundimagetotheGPUasanormalRGBAtexture.However,thevideoframescomingfromthecameraareintheYUVcolorformat,whichisnotdirectly“understandable”fortheGPU.WewillneedtodelegatethecolorspaceconversiontotheGPUaswell,anduploadtheYUVencodedframedatadirectlyasatexture.AgenericRGBAtextureisnotexactlysuitableforthispurpose,becausetheY,UandVchannelsarepackedinseparatesectioninthedatabuffer,andeachcolorfulpixelshouldgetdatafromeachofthethreesections.Weneedasimplearrayrepresentationofthedata,andthemostappropriatetextureformatforthispurposeisapuremonochromeformat,whichiscalledLUMINANCEonlytexture.DataarrangementinthedifferenttextureformatsNowthatwehaveboththebackgroundimageandthevideoframesastextures,it’stimetoexaminehowtheGPUwillprocessthisdatatocreatethedesiredvideoeffect.FragmentshaderWekepttheimplementationoftheshaderasclosetotheoriginalJavaScriptsolutionaspossibletokeepitunderstandable.ThemostinterestingpartistheYUVtoRGBAconversion.ThefollowingimageillustrateshowashaderprogramprocessestheYUVformatdata.WecanseethatinspiteoftheRGBAformat,technically3differentimagescontainalltheinformation,andeachpixelreceivesluminanceandcolorinformationfromallthreesub-images.BecausetheUandVchannels’heightandwidthresolutionsarehalved,theevenandoddrowswillcontinuouslyappearinthebufferwhenitissampledwiththefullresolution.Wehavetolivewiththissmalldrawback,becausewehavetrickedtheWebGLsubsystemto“think”weprovidedamonochrometexturewiththefullresolution,butinrealitywesentafullresolutionmonochromechannel(Y)andtwohalvedresolutioncolorchannels(UandV)inthesametexturebuffer.Intheshaderprogram,thefirstfourdefinitionsareusedtosearchfortheYUVchanneldata.Therestshouldbeprettystraightforward,sinceit’sreallyanalogoustotheJavaScriptversion.VideostreamtransformerWearenowatthelaststeptogetaworkingtest.Weneedtomodifythetransformerfunctionfromourpreviousversionssothattheapplicationuploadstheframesastextures,thengetstheresultsbackbeforecreatingtheoutgoingvideoframe.FirstattemptWearegoingtoreadtherenderedpixelsasaframe-bufferwiththe“readPixels”function.Theapplicationworks,buttheperformanceisfarundertheexpectedlevel.EvenonahighperformancePC,theframeprocessingtimeisaround10ms.Afterinspectingtheperformancemetrics,wequicklyfindthatthe“readPixels”functioneatsuptheresources!Wecandobetter!SecondattemptBycarefullycheckingtheVideoFrameconstructor’sdocumentation,wefindthatitcanacceptcanvaselementsaswell,soalsotrytheversionbelow.Thefollowingchangesareneeded:🎉Hurrah!🎉Thespeedhasincreasedalot!Now,theframesareprocessedin1msonthesamePC.Excellent!!!BenchmarksTocomparethedifferenttechnologies,Ihavemeasuredtheframeprocessingtimeondifferentdeviceswithalltheoptionsdiscussedinmyseriesofarticles.AllofthedevicesrunChromev97.TheresolutionwassettoHD(1280*720)FrameprocessingtimeAt30frame/sec,theavailabletimebetweenframesis33ms.Inthespreadsheetabove,theredvaluesindicatesthecaseswhentheframeratedroppedduetoextendedprocessingtime.Itsobvious,thattheGPUprocessingisfarmoreperformantthananyothersolution,andiscapableofreal-timeimageprocessingonslower(mobile)devicesaswell.AnotherinterestingresultistheperformanceofWebAssembly,whichseemstoberelativelyfasteronslowerdevicescomparedtoJavaScript.AppendixUsefullinksSomeofmyfavoriteWebGLresources:https://webgl2fundamentals.org/https://thebookofshaders.com/WebGLinitializationscriptYoumayfindthedetailsinteresting…MorefromDoclerEngineeringGetinsightsfromDoclerEngineeringteamsonhowwearesolvingdifferentchallengesinlivestreaming,softwarearchitecture,infrastructureandtechnicalmanagement.ReadmorefromDoclerEngineeringRecommendedfromMediumfloridaelderlycareorganiz//platform.twitter.com/widgets.js fromTwitterhttps://twitter.com/preferredpcareJustinShermanReactCodingInterviewChallenge23AbhinavGautamReact-RouterQuirksJohannesBauminBetterProgrammingHowToCheckifaPropertyExistsonaJavaScriptObjectfloridaelderlycareorganizThepositive#businessimpactof#mindfulnessatworkhttps://t.co/GiUQ8oE9GJby@MindfulnessNCLGabrielDinizBuildingaJavascriptScrollingCarouselAaronHedquistX-RayGlowShaderEbonyHargroObjectOrientedProgrammingw/JavaScript|“this”,“new”&constructorsAboutHelpTermsPrivacyGettheMediumappGetstartedSzabolcsDamján21FollowersFollowMorefromMediumJanHalamainSuperfaceReliableIPGeolocationwithSuperfaceDavidLueckeinTheFeathersFlightpathWhat’snextforFeathersin2022?michaelsorenseninITNEXTHowtoavoidNPMsupplychainattacks.MiroslavBodečekinDappetizerIntroducingDappetizer:anIndexingFrameworkforTezosHelpStatusWritersBlogCareersPrivacyTermsAboutKnowable